Uh, no, the gender of the main character can make a huge difference in any work, good or bad. Limiting ourselves to literature: Would Othello or A Farewell to Arms be anything like the original story if the main characters were female? Would Medea or Hedda Gabler make any sense with male main characters?
Besides, even major details that don't affect the plot matter in a story. Setting, for instance, need not be a plot device, but it is a major contributor to a story's success. For instance, I was writing a story set in a small town, and I wasn't getting anywhere with it; it seemed very bland. I decided to set it in a walled city instead. Even though the change was superficial, it caused a major difference in the atmosphere of the story, which greatly improved it in my opinion.