I think a departure novel would only become a problem after you have had enough success with the public to be pigeon-holed as a [insert the genre] novelist. A nice situation to be in, actually. Or is it?
Just for fun, let's speculate. John Grisham has had some success writing humorous nonthrillers, so obviously his fans were willing to follow him there, and probably he attracted some new readers. But what if he decided to write a high fantasy? There's a much bigger departure. I'd expect his base of legal thriller readers to be dubious. I'd expect fantasy readers to be dubious. I'd expect the critics to be sharpening their knives and licking their chops in hopes of a spectacular failure.
But because he's John Grisham, the dubious and the hungry would likely give his high fantasy a glance. If it's good fantasy, the fantasy readers would read on. A significant number of the thriller readers might drop off -- this isn't their kind of thing, and author-loyalty could only go so far with them. They would be the bigger problem, I'd guess. But would they boycott Grisham forever? Nah, I'd guess most would be in line for his next thriller.
Grisham could use another name for his fantasy. But then he'd probably end up like Stephen King when he used Richard Bachman and found out how important the name Stephen King was. That is, a first fantasy novel by Grisham would generate a lot more interest off the block than a first fantasy novel by Johanna Grizwold. Are the publishers going to print a small run of Grizwold, or risk a Grisham-sized one? What about plans for a strategic leaking of the truth?
Heh, this is starting to sound like a good industry-insider novel. Grisham is struck by the truth on his way to Damascus (Virginia) -- legal thrillers suck; he must write only the sacred fantasy from now on! He does, but badly, but he can't see it, and turmoil ensues!
Anyhow, if thrillers are your real love, start with them. If you're lucky enough to score a hit, it's another thriller the publisher and fans are going to want.