I get why you don't like it, but it's sort of inevitable with bigger fandoms. like what everyones saying most books dont have large fandoms unless they reach HP/Twilight status, so you probably don't have to worry.
personally i'd kind of like it with my work, but to each their own