Aren't there different kinds of difficulty? If we mean physical exhaustion, writing isn't "hard" in the way that shoveling coal and baling hay are. But, on the other hand, the purely physical tasks don't engage more than a fraction of your brain and don't lead to worry or doubt that what you have done is worthless, the way writing can. And at least when you have baled a bale or loaded sixteen tons of Number Nine Coal (whatever that is), or dug a nice neat ditch, you know you have accomplished something positive. When you felled a great tree, you knew you were a lumberjack and you're OK after sleeping all night and working all day. Etc. But when you have written a chapter or a stanza it may be rubbish, or you may have to tear it up and do it again as your work progresses.
Good writing, at least, is certainly work, IMHO, and often very unrewarding in any sense.