When you grow in a racist, patriarchal, homophobic, classist, sexist culture your way of thinking becomes infused with ideas that are necessarily counter to freedom and basic human respect. Even the most conscious amongst us will continually revert to patterns of behaviour, thought, or speech, that are counter to our stated beliefs. Due to a constant desire to privilege our experience and our existence over another often we do not even recognize these lapses.
In other words - it's less mustache-twirling villainy and more about the banality of evil. It's less a matter of individuals deliberately going out of their way to making other individuals miserable and more a matter of people being complicit in an environment that harms the Other (however that is defined) in both overt (legal, economic) and covert (psychological, social) ways.