no simple answer but lots of passion
Typically the rule is "if you want to fictionalize, don't call it a memoir." Having said that there are a number of areas in memoir that cross the thin line between fact and fiction.
For one thing, most dialog, even when repeated from an hour ago is only an approximation. (Try recording a conversation and then reciting it word for word 20 minutes later. If you're like most people you will misremember quite a few words.)
Also it is widely agreed you can distort names and identities in order to protect privacy. (Just put the disclaimer in the beginning that explains you are doing this.)
I suppose if you have an intense need for describing rooms and clothes you could invent visual details. If you really want to describe what a person is eating and are not sure of the exact dish, I suppose you could make it up. But those nuances are beside the central point. The whole idea of a memoir, in my opinion, is to tell the best possible story from your own authentic recollection of events.
And that is not really a limitation. In fact, one of the neatest things about memoir writing is that it is an investigation into your own life. By writing, researching, interviewing, visualizing, and layering, and so on, you will eventually remember (and understand!) a lot more than when you start.
Jerry