"All indications of his skin colour are from characters in the book, and none of them can say with any certainty he is not white."

That's not really what I take away from your examples, though. If people are asking, they're asking because they're pretty sure he's something other than white, otherwise they wouldn't be asking. In fact, the examples you cite suggest that if they're sure of anything, they're certain he's not white and they're trying to ascertain what that "other" is.