As is often the case, your poem instantly brought a memory to mind. Being me, I'll share it.
I once ran into a guy who'd pursued me romantically years before at a job. He gave me the creeps and I turned him down; he was rather ugly about it and said numerous (false and salacious) things about me to our mutual colleagues (and alas, those who didn't know me well were not unwilling to believe him). I moved on to another job shortly thereafter and forgot all about him -- until I ran into him at a social event.
Because it had been a few years, and because I suck at associating names and faces, I didn't instantly place him (plus he'd lost a lot of hair and put on some weight). But as soon as he spoke, it jolted through me who he was (and it was indeed a jolt). Your last line pretty much describes the feeling I had -- the world drained of color. (And I must admit I bolted like a cannon from the event.)