See, that's just it. If you don't mean it in a negative way, then I don't hold it against you. If someone else is offended for a legitimate reason (legitimate reasons include "your statement is one that further emphasizes hurtful stereotypes that I face every day," but not "ohmigawd you don't even know I have like 7 gay best friends and my sister is black and actually I've never met an Indian but I'll bet that what you said would hurt their feelings don't you know anything????"), it would be nice if you're willing to apologize. If you're not, then be prepared to be blamed. If they're just there to guilt you so that they can feel better about themselves - if they're only "insulted" because it's fashionable to take every little slight personally, no matter who the supposed victim is - then really, you've nothing to apologize for.