The two fringe groups were not Nazis and anti-nazis. They were Nazis and communists. The communists had their own agenda, they were not simply anti-nazi.
There was also a large group of more centrist liberals that were more focused on stuff like trade unions and public sector unions. Hitler realized he could work with this group as long as he agreed to meet their short-term political goals.
And to your point, the KPD, along with their militant wing Antifaschistische Aktion, attacked the SPD far more aggressively than they attacked the Nazis.
