Gender affects averages but not individuals as much. You might find more people of a certain gender liking a particular genre or writing style, but that doesn't mean people of the minority gender among a fanbase are any less dedicated to the genre. (Wow, lots of use of gender and genre here. I remember Weird Al making fun of Eminem for talking about an "entire gender of music.")
For example, you might find fewer men working with romance (just a guess on my part), but the ones who do are probably just as likely to work out as women who work with romance.
Some individuals who have trouble working with members of a certain gender can obviously be problematic, but that's an individual thing and not usually a problem. (If you work best with a particular gender, it would probably help to have an agent of that gender. If the agent works better with a particular gender, it's best if you're that gender. But with most people, it just won't be significant enough to make a difference.)