No there are more than three, otherwise I would have used the word "few" (I said "many" meaning more than several).
The ADL reported that there were over 40 subsets and hundreds of chapters around the country, with a membership of about 5,000 people. But the ADL also notes that the organization has experienced a surprising resurgence (some now estimate the organization to be approaching 8,000).
Christian Identity (they believe that white people of European decent are the "lost tribes" of ancient Israel) is another group that has been affiliated with both the KKK and neo-Nazi groups. They number over 25,000 (some estimate up to 50,000).
When we look at "hate groups" we have to be careful (the Southern Poverty Law Center lists the American Family Association (AFL) as a hate group for its position on sexuality, and the Tea Party as a hate group for "general hate"...
). That's why I've stuck mostly with the KKK here. But I think that we'd be safe to include the skinheads, neo-Nazis, Christian Identity, and the Aryan Nation.
Again, I never made the claim that these people are the majority, or even that they are a substantial minority. I said that there are many racists out there who hate President Obama because he is black. Even if we only considered the KKK, 5,000 to 8,000 is many.
What is interesting is that while the KKK has grown between 2014 and 2015, other white racist groups have declined. But black racist groups have increased in that 14% rise over that year. If you look at racism (to include "hate crimes"), it is on the rise not among white people but with African-American racist groups.