There are plenty of permanent firework stores, but since it's a very seasonal business, they will probably be outnumbered by outdoor tents around the 4th of July. You can buy individual novelty items (like tanks and battleships, flying UFOs, or exploding Osama bin Laden heads), bottle rockets by the gross, or prepackaged variety packs (although that's not as efficient). Also, around the 4th of July a lot of grocery stores will carry some of the variety packs. However, buying at a dedicated fireworks store will get you a handful of "punks," which are slow burning sticks, similar to incense, that you use to light fireworks so you don't use a match for each individual piece you shoot off (That's what I called them growing up in Missouri anyway; I'm not sure if that's regional).
Also, even small, relatively rural towns will likely ban fireworks within the city limits, with possible exceptions for the first few nights of July.