I've been scared of that, too, when I've seen fics in my smaller fandoms that got no response whatsoever (I reviewed when I liked anything about the fic). But I wonder how much different it is to enjoy writing a fic and never post, and enjoy writing a fic, post, and get little to no response. Either way, as long as I enjoyed it, its primary purpose was achieved, even if I'm a little disappointed. I might give it a shot, stopped avoiding little fandoms.

I think I'd only be truly crushed if I got no hits or faves or reviews posting in a large and active fandom, on a popular site, writing a popular character or pairing. I'd endlessly agonize over how I personally managed to fuck that one up.