I think there is also the risk that by answering the question "Who is your agent" the person who asked may then send off a submission to your agent using you as a reference, as if you are recommending them as an author. Which is . . . tricky . . .
But I for one actually often (not always) say who my agent is and I'll always share who my publishers are (Scholastic UK/Commonwealth, Weinstein Books in the states) - with publishers it's just good advertising. I think most others do here as well actually. There are a few exceptions, Cheshire Cat really wants her anonymity for example.
I do think unless a very good reason is given, that the authors who refuse to share this information do tend to have something to hide. You see it all the time, an author suddenly appears out of nowhere with some advice for us all along with the news that she "has signed with a TOP agent and publisher", but then after further questioning they get quieter and quieter. The fact is that agents and publishers love announcing book deals to the world because they want the buzz to start ASAP. Any author who tells you that they don't want to share that information because of something in the publishing deal is very suspicious to me. Now if they don't want to share the information for the reasons Karen said, that's different.
Anyway, just my thoughts . . .