Truth be damned!
Per Stijn: Libel is difficult to prove and more difficult to win damages. I just asked this question on another Board when I found a story using my real name in a porn story that did not happen. I was told I'd have to prove the writer had the intent to damage my rep, and that the damage had a real impact on me financially, socially or otherwise. Since I can't disprove a negative, (that the story did not happen), and since I can't prove the writer intended to hurt me in some way, and can't prove I was damaged, the chance of winning such a case is virtually impossible. Remember the Falwell vs Hustler case when the cartoon Larry Flint published was horrific and obviously false.
Per pollykahl: There's no law that says memoirs have to be true to the n'th degree, is there? In fact, It's almost a 'given' that most biographies are a glossy version of reality and some, if not most, play fast and loose with the facts.
Even those people who have been in the public eye and therefore subject to the memories of the public, take liberties using 'spin' or outright changes in the 'facts.' Is it absolutely crucial to the point of the story that you must quote a specific person exactly?
I don't see any reason why you can't enhance the story as we all do when we're telling 'a whopper' to our friends.
You are a writer, right? So write.