People do, regardless. There's not much you can do to stop that, unless you explain directly to them that you have severe anxiety.
And, even then, people will STILL talk shit about you because they either think that it makes you dangerous / autistic / whatever, or that you aren't "trying hard enough."

You're kind of fucked, regardless. Much like having any kind of mental condition.

It's a shitty position to be in.

But it doesn't mean you need to promote the negativity.
Otherwise, you're kind of just being an adamant raging douche.