But all the publishers I queried were ones that accepted unsolicited submissions. Now I let my agent do it all
That's the only way I would consider querying agents and subbing to publishers at the same time. Because agents get a commission on the sales they make for you, it's in their best interest to go after the bigger, advance-paying, large-distribution publishers (who, with a few exceptions, usually work mostly through agents). It's not worth an agent's time to sub to small houses that don't pay an advance and don't make many sales.
So if you sub to those small houses, you likely wouldn't be stepping on a prospective agent's toes. However, do you want to go big or go small? What happens if you're subbing both at the same time and you get an offer from a small house? Do you rush to accept it? Or do you wait and try to get an offer from an agent who can get their foot into the door of much bigger houses than you could on your own? You eliminate that dilemma by querying agents, first, and then, publishers (largest to smallest). That way, you don't burn any bridges or have to decide between the small bird in the hand or the potentially larger birds in the bush.
Some people come at the idea with the assumption that if they get a publishing offer first, it's easier to get an agent afterward... but most successful agents won't want to spend time on very small publishers, so it's not as much of a carrot as you'd think.
ETA: I'll add that it really depends on what you want out of publishing. If your goal is just to be "published" and you don't care if it's with a really small, e-pub only house, then you'll probably have faster (but less lucrative) success trying smaller houses that take unsolicited MSes. If what you really want, though, is to have an agent and publish with a larger house, then maybe you don't ever sub to small publishers. Maybe if this book doesn't attract an agent, you trunk it and write a second or third book that does. Or maybe you publish with a small, decent house that gives you some editing lessons and helps you improve enough to go after bigger fish. You decide what's important to you and plan around that.