It comes from people who read ACD without taking into context the historical period and attitudes of the time. Holmes isn't nasty with women nor is he a misogynist by the standards of the time. He just doesn't see the point of a romantic relationship for himself. He also teases Watson about his marriage, and a lot of people miss that he's ribbing a colleague, not making a blanket statement about marriage being a waste of time.