Okay, there's an agent that was really interested and wanted to see a second round of revisions on one (hasn't even seen the one with the offer yet) but I never got back to her on the next round of revisions yet, so she would be the one to contact, yes?
I would contact her, yes.
Even if there isn't an advance involved, if you would prefer to go into the contract with agent representation, it's worth contacting them...with a subject like that says "Offer in hand from ____.".
I actually had such a situation. In the midst of querying agents, I got a two-book offer from Kensington for their digital-first line, which doesn't offer an advance (although, if they decide to put the books in print eventually, the print edition will include an advance). So I updated agents who had partials or fulls...and I contacted two agents who'd rejected partials/fulls but whom I still really wanted to work with. One of those agents did reconsider.
And I ended up with three offers of rep. It was quite a whirlwind week.
So I'd say it's worth seeking an agent when you have a pub offer on the table. And i'd even say it's okay to contact agents who've rejected your partials/fulls. The worst that can happen is they'll say no, which would be understandable. Ultimately, I really wanted an agent because I'm looking ahead, not just at this book deal but at a writing career and felt an agent would be an important guide. (And already i think she's worth every penny!)
But there are lots of authors who've taken contracts without an agent--if you do, I recommend you have an attorney with literary contract experience review the contract before you sign.
In terms of timing, I had a limited window specified by the pub. I explained that I was in the middle of querying and needed to update agents about the offer. The editor wanted an answer within a specific timeframe...so I made the timing clear to the agents when I updated them.
Good luck to you! Keeping my fingers crossed for you!!