Listening to someone saying it is as good a starting point as any. Say it yourself, perhaps? Then write it down.
Hey bufty, as Al and Grunkins have said after your post, I always read that numbers of any type, are written out as words, when in dialong.
But, following your train of thought, if I say aloud to myself, "I will be there at 5:40" ... that is what I hear, the number 5, colon, and the numbers four and zero. I mean I know I don't actually "hear" that, but in my mind I visualize a digital clock.
When I write dialog, unless it is an exact hour (which I tend to write as "number o'clock"), I find that I strongly want to write the digital clock version of time.
I was hoping to find a thread where others have discussed this. My big question is if I 'break norm' and write the digital clock into my dialog, how offsetting is that to average readers? & If I do this, should I also do it for the even hours as well?
Ok, now that I have written this out, I assume that I will have three posts pointing me to other threads.

doesn't it always work that way.
Thanks again!