Would the same go for "backward" and "forward" and "downward" and "upward?" I always write them with the "s" the first time, as that's how I hear it, but then change them back in the final telling.
I've never heard anyone say, "Hey, you're working A$$ backward." The "s" is usually always included in dialogue. And I've had this said to me many times.