I think it depends on how big the fandom is. The bigger the fandom, the more likely there is a silent majority. However, in a small fandom that's dominated by one juggernaut ship, it definitely stands out.

Fallout fandom, for instance, definitely has this problem.