Actually, all the stuff about shadows skin colour, in the start of, and most of the book, comes from people (the first of which being a racist prison guard) all /ASKING/ (Implying the people physically looking at him aren't sure enough about his skin colour to definitive state it) if he has any gypse in him, if he has an native American in him, etc some one actually used the "cream and coffee" expression in front of him about someone else.
All indications of his skin colour are from characters in the book, and none of them can say with any certainty he is not white.

I think I could forgive someone for thinking he is white.