Why do you think things like affirmative action or preference shouldn't be going towards people who lack white privilege? If they are at a disadvantage naturally because of the color of their skin, is it not fair to create something to help make their life easier? If some kid grew up in a bad environment due to his socioeconomic status that stems from centuries of oppression to his ancestors and still effects the person to this day, should the person not be given more opportunities when there is some promise shown? I don't see an issue with that.
And besides that, to what degree do you think it's unfair that they may have SOME more opportunities? Mind you, there aren't too many. Affirmative action is one, sure. There may be a few more here and there. But life is definitely going to be much, much more difficult as a whole. Besides the fact that not even that many people will even benefit from those potential opportunities they have, it still doesn't outweigh the benefits of having white skin.