Glaciers grow because snow falls on them and adds to the ice already there. As mentioned, this is, in terms of human lifespans, a pretty slow process. In other words, no glacier will rise up and swallow a village built next to it.
More critical would be where, in relation to the glacier, the village is.
We have all seen the ice falls at the end of glaciers, with those in Alaska's Glacier Bay, near Juneau, to be some of the most well known. Big chunks of ice falling off and becoming ice bergs, etc. People would figure out really quickly not to build houses there.
But there are, and no doubt will be for as long as people think it's cool (pun definitely intended) to live near the giant rivers of ice, human habitations beside glaciers. This zone is where, in summer, some ice melts, provides source of water, and as with most boundaries between eco-zones, an abundance of life, both flora and fauna.
And if you consider the northern populations on this planet, they live on the tundra, which isn't all that different from living right on top of a glacier.