Maybe you could approach your worldbuilding a different way. Instead of thinking of it as (basically) stage-dressing and props and backstory, think of it as the entity that made your characters who they are. We are not just products of our genetics; we are products of the world we grew up in, from the government system in place down to the neighborhood we lived in. So much goes into shaping who are: culture, religion, relationships, politics, poverty, wealth, physical environment, war, peace, technology, geography...
What shaped your characters? How can you let it influence how they think, speak, and act? How can you show that?
I was going to suggest something similar.
For instance, readers can learn a lot about how religion works in your world from the attitude different characters have about their gods and from the little rituals they engage in and so on.
You can portray a lot about the economy from the things your characters have access to and their attitudes about them. And you can get social norms across by the little things people take exception to or tease one another over.
And magic, if you have it in your world. What is it used for and how do different people think about it? What does it cost its users? This can be shown in action, rather than told in narrative outtakes.
When world building falls apart in some fantasy is when things become incongruous. For instance, a world where religion seems to be institutionalized and important, yet none of the main characters give it so much as a passing thought,
and this isn't an act of rebellion or a conscious rejection of the status quo. Or a seemingly medieval world with peasants and nobles and knights, yet there's a robust middle class that lives better than lords did in the real middle ages, with china tea services and sugar and silk underthings and homes that are far nicer than any medieval castle. This would suggest the presence of a very developed level of trade, if not active colonialism.
Or a world where magic is powerful, and fairly ubiquitous, but everyone thinks of it as something of beauty and wonder and never as a weapon or a menace (except on the battlefield). Think of all the fear and disagreement about firearms in our own world (because they can be hidden and kill quickly and at a distance). Mages would certainly be scary. People have been scared of magic in many cultures that believe in it, even when it was limited (in their perception) to soothsaying and ill wishing. Imagine if someone could call fire from thin air or blast someone with a flick of their fingers, and imagine if such talent were fairly common.
Of course, some of it depends on the type of fantasy as well. Whimsical, fairy-tale worlds aren't held to the same standards as stories set in more serious, internally consistent settings. And some fantasy elements are beloved because of the aesthetics or coolness of them, rather than their realism or internal consistency.
