There are five or so big publishers left, and hundreds of respected middleweights, plus the specific international markets, plus subsidiary rights. There's a lot of work for agents, considering how big the playing field still is.
Agents have taken on the bulk of gatekeeper functions as far as reading slushpiles. I know several agencies receiving an average of 10,000 to 20,000 queries a year, all of which they and their assistants at least glance at. Many large publishers no longer take unagented, unsolicited manuscripts from writers. Of those who do still have slushpiles, the wait times on those submissions can be months or years.
(Some authors bypass this and self-publish, but bad writing is bad writing anywhere it ends up.)
Then consider genre: good agents, like good publishers, tend to focus on a few types of books in fields they know and like. A nonfiction agent may know lots about that market, but nothing about science fiction or romance - so it makes sense not to represent those genres.
In fact, too-broad representation of many genres and categories, from a literary agent or a publisher, can be a red flag indicator about their general competence, business models, and effectiveness at their jobs.
