I think you're really underestimating how much of an effect those factors have on how we see/react to our surroundings. If you grow up in a racist society, you're hard wired to be racist, and it's very difficult for most people to recognize that and transcend it. It's nearly impossible when you're stupid.
Why would someone stop hate mongering if they don't see what they do as hate mongering, and if nearly everyone and everything around them confirms all of their biases on a daily basis? Plus, most racism isn't an active thing... if you're a black American, you're presented with an ideal of beauty that's very much caucasian. You're told that the way you talk is flat out wrong. These things obviously effect the way white people see black people too.
I think saying, "Just change how you act!" is overly simplistic. I wish it were that simple, though. Don't get me wrong!
(What was this thread about, again? My views on racism, right? Oh, good.)