I don't think it's really on you to be responsible for who follows you, if someone really wants to get an idea of who they follow on tumblr they can easily go through that person's archives and see what fandoms they post for. If that's not your fandom then I'd imagine you don't have many posts related to it, so if that's really their only reason for following you then it's on them for not researching what your blog is about.

People on tumblr follow and unfollow constantly, I tend to be more picky about who I follow because I don't like to unfollow unless I absolutely must (because I know it can hurt to lose followers.)