I'm percolating an idea right now that I could develop into a short story, then into a longer version that would be the first episode in a series of novellas.
How long a period does a publisher usually sit on a first rights work?
That depends very much on the publisher. Some will want rights for as long as the piece remains in print, others give a fixed time. Others expect all rights in perpetuity (and no, that's not good for you).
And does the novella even count as a reprint of the short story?
Sort of, but not exactly.
Again, it depends on the publisher and their contract.
What you also have to consider is what the novella publisher will consider a reprint, and whether they'll be willing to publish something which has been previously published in another form. Of course, you can't look into the future and know who might want to publish this, so it's tricky to work this one.
If I submit the short story, what limitations does that put on selling the longer version? If the short story is under a first rights arrangement?
It all depends on the contract and the publishers concerned. (This is getting boring, right? Sorry.)
Some publishers will insist that you can't sell rights to any derivative works for a year or two, and this would definitely be a derivative work. Some publishers won't want to buy a derivative work. Selling first rights to something doesn't always restrict subsequent rights sales but it does usually, and a second publisher isn't likely to be keen on publishing something that another publisher has currently in print.
Or, any other arrangement beyond an "all rights" (I wouldn't submit under an "all rights" thing)?
Why wouldn't you agree to selling all rights? So long as the contract has appropriate reversion clauses and you'd be paid appropriately too, there's not necessarily a problem here.
Finding the rights/licensing thing a little confusing.
Rights are confusing.
Roughly speaking, you can't sell the same thing to more than one publisher.
You can't sell first rights more than one time.
You have to tell publishers if a piece has already been published.
Sales of derivative works are often restricted in a contract.
Don't sign any contract until you understand completely what you're signing away, and how you can get it back if you need to.