Just don't be Robert Jordan and spend three pages describing buttons on a dress or whatever. Please don't. 
True. Don't go overboard... unless the scene calls for some young girl to be fascinated by clothing at some event, store, or encounter. Or young guy.
I know a lot of readers who skip those kinds of details. They still love the books, but when it comes to red samite gowns and rustic soups of onion and beans, they just start skimming.
As a writer, I'd rather put in the details and have my readers skip them then leave the details out and have people tell me I need more.
If you examine the history of any aspect of human society you'll find that common sense has nothing to do with it. People do all sorts of odd things that make their times and places stand out.
The inter-relationship between culture, natural resources, human needs etc is quite complex and gives room for the uniquenesses that can make one fantasy novel stand out as not being the product of generic thinking.
If you haven't read it, I recommend Diana Wynne Jones' A Tough Guide To Fantasyland. Amongst other things her entry on Stew shows what happens if you don't think through matters like food and clothing.
It's one of those balanced craft things.
I've never been very fond of stories that stop dead to describe an outfit in great detail.
Does it matter? Do you need to describe the clothes at all?
In my fantasy novel I only mention clothes when it's something out of the ordinary - a lady who usually wears skirts wearing riding pants, someone from a different culture wearing their traditional dress (or undress, one of my cultures has the guys go around shirtless). Most of the time, I just assume they're clothed and leave it at that.
Some things to think about regarding clothing:
1. What is available to your people? Plant fibers tend to dye poorly. Animal fibers better. Silk best still. Animal fibers tend to be very warm to wear, though. Egyptians got around hot weather + lots of linen by making lovely beaded over-dresses. Minoan, women, though, wove colored wool into intricate patterns. No matter what people are wearing, some industry needs to support it. If you're describing a rugged frontier society where everything is self-made and there are plenty of sheep but nothing else...then someone shows up in a silk dress, there are problems. (Or they have cotton, but vivid patterns - it's hard to create those on cotton as it dyes poorly.) OTOH, if you have trade coming in from everywhere, I can buy vivid colors + all kinds of fabrics.
2. How much stratification is there between the rich and the poor? I can totally buy even a low tech - but highly stratified society - having some fancy clothing. If your world is low tech + highly egalitarian, though, people probably *aren't* going to be wearing tons of incredibly intricate clothing (although there may be some fancy embroidery - people often do all kinds of things during their time off to make their every day objects pretty). Also, your every day people probably aren't going to have very much clothing. Anne of Green Gables takes place in a post-industrial society but she still has three dresses. THREE. This is because clothing is quite expensive. If you have a high queen with millions of subjects, sure, I can see her changing her outfit several times a day. But your every day people probably have several shifts + maybe a nice dress, an okay dress, and a dress they wear pretty much every day. Your average peasant does not have a massive wardrobe.
3. Cutting cloth wastes some. From above, remember that cloth = very expensive + time consuming. This is why a lot of garments (i.e. togas, chitons, tunics, etc.) are pretty much one size fits all + don't involve a lot of cutting. Tightly tailoring clothing appears to be something that primarily happens when technology makes cloth cheap enough that people are okay wasting it.
4. If your people are working, they're probably wearing clothing that they can work in. IF they're not, they may well be showing off how they *don't* need to work. Examples of this range from women staying out of the sun to remain very pale...to long sleeves, foot binding, high platformed shoes, enormous shoes, huge skirts, etc.
Clothing is more than protection from the elements. It's the chief way an individual can show his or her place in a society, and by adherence to social convention or defiance of it, how he or she views that place and the society as a whole. The extent to which a society regulates clothing is also telling, and the cultural level of a country will show in its tailoring and decoration, or lack thereof.