Yep, it would never occur to me that when someone said "It's the size of a football field" they meant that it was the size of a football stadium.

However, I do think that FennelGiraffe is right in that most people's mental image of a football field includes the end zones.

It's pretty approximate, anyway, which is why it's useful (or not). When someone says "I couldn't hear him, but he was a football field away," I get a very specific mental image of the two people standing on the opposite ends of my high school's football field.