When I was a kid (oh, starting when I was probably 6 or so) we called the 'harder' books chapter books. That is, the books whose pages weren't taken up mainly by pictures, and that had big long sections of just...words. For us, saying we read chapter books was like bragging.
Once you get past The Magic Treehouse etc. they're not called chapter books anymore. I'd say that if the people reading it are mainly over ten years of age, and it has no illustrations at all inside, even if it is separated into chapters, then that isn't a chapter book anymore.
Actually, now that I think about, most MG books are probably considered chapter books. Or at least, that's what we called 'em.
Hope my little reminiscence of being six was somewhat helpful. ^.^