I doubt a book is rejected as "too smart." Note this isn't coming from a crotch-scratching, overall-wearing hillbilly, I am a crotch-scratching denim-wearing PhD biologist. That said, "too smart" is bullshit. Stephen Hawkins wrote a fucking best-seller.
There ARE books written too dense, with too many asides and info-dumps, etc. to match their target audience, and there are plenty of them. But saying they were "too smart" is inaccurate, just as saying a $200K rocket-car was rejected for being "too efficient." If you fail to match your market (or the publisher's idea of it) you can be rejected, but that isn't a matter of "too smart" so much as not meeting certain genre standards, etc......you write Harlequin Romance or pulp horror with an advanced English college-level reading level, you'll alienate a shit-ton of your audience. Because they don't have that reading level. But I would say if anything that isn't you writing too smart, it is an author writing too dumb--write what you feel compelled to write, but if you simply won't match your market, that isn't too smart at all.
You can write beyond your market, but that seems to be an issue on the author, failing to meet their market, so I have difficulty calling that "too smart".....it would be your fuckup for making something your target audience does not want, and the bottom line is this: written work, if you intend to chase commercial success, IS a commodity--you either provide what the customer wants, or you do not, but to lament the customer not buying what you have if it isn't what they're looking for is silly at best, no matter if you're selling shoes or lobster or thrillers.