It makes me sad, too. Reading broadly through a lot of the "classics" has taught me a lot about what I do enjoy in a book. I'll admit to not being a fan of Henry James or James Joyce, but I've read them, and can articulate why, and it sheds a lot of light on why I enjoy the sparser, cleaner prose of Hemingway or Graham Greene. It's an amazing thing to poke around in the Canon, read Tristram Shandy and suddenly understand Joyce better.

There's no shame in putting down a book you aren't enjoying, but there is something sad about not being willing to try something and broaden one's horizons. Not everything has to be Great Literature, but not everything has to be Harry Potter, either (and I freely admit to enjoying both).