Is it okay to query two publishers at once?
Absolutely! Keep querying as many as you want until someone buys the book! Keep it reasonable, though. Start with five or six and once you hear back from most of them, send out another batch. Remember that publishers (editors really) are as much subject to taste as you are. What one hates, another will love.
And if a book is out to publishers, does that make an agent not want to consider it?
This is a yes and no. If a publisher makes an OFFER, an agent will want to see it. But if what you want is an agent, then you should try to get one BEFORE you start searching for a publisher. See, if a slush reader at the publisher didn't like something and rejects it, it'll be more difficult for an agent to contact the editor the slush reader works for to convince them that the editor WILL like it. After all, the slush reader is supposed to have the same taste as the editor. That's the point of hiring them as a first reader.
But, in reality--if the book is good enough, it really won't matter. When I was querying publishers for our first romance, it was rejected by the same company (Tor) that later accepted it when submitted by our agent. It can happen.
Personally, I'd say query a whole bunch of AGENTS to start, rather than publishers. There are a lot more doors open to you with a good agent (as many publishers are moving toward "agented submissions only.")