I love all the actors and actresses in HH.
The only other fans I know IRL are like ten-years-old or less (lol what do you mean I'm not the target audience?), and everyone loves everybody, so I really don't have the same problem you do.
I'm aware of how the tumblr fandom operates, however, so I just mostly stay away.