I've found it to have a huge benefit. When I was co-writing a novel, we got some critiques on it. The critiques were rather odd because all the comments pointed to rather non-specific things. Most people would have just corrected everything without really understanding why they were getting the comments, but I wanted to know why.
And I got my answer because I critiqued about eight other first chapters. Not only did I look for problems in other people's work, I asked why I was finding this a problem. Over the course of those eight critiques, I saw my story through these other stories' eyes, and I finally realized what those non-specific things pointed to it. Every writer's work suffered from exactly the same problem, and the more questions I asked myself in the critqiues, the more I started to come up with possible answers.
Remember, a critiquer is only as good as the critique. Some people say, "Just tell me what's wrong, and I'll fix it," and they don't realize that this problem might point to a much bigger one that the critiquer can't quite process. For example, if you have forced dialogue, is it just an issue of writing the dialogue better or could it be that there's a problem with the story itself? A critiquer might see the forced dialogue but not catch the story issue. But if you went through and critiqued several other stories with your eye on dialogue--even when it works right--then you'll begin to process the answers to your own questions.