I just had people with infidelity squick in mind. Personally, I would think the whole forbidden love thing between Bates and Anna as more of a draw for audiences. It's one of the more intriguing relationships in the show.

Although, my mind still doesn't completely trust Bates for some reason. I feel line there are still some unanswered questions when it comes to past, and his indirect but possessive response to Molesley inquiring about whether or not Anna was single made me raise an eyebrow, especially since he IS still married. (Aww, heck. I still ship it though. lol)