I think a good example of your situation would be in Harry Potter! J.K. Rowling uses minor British swearing. "Bloody hell" is about as bad as it gets, until one battle scene when a frumpy witch mama yells, "NOT MY DAUGHTER, YOU BITCH!" I would say I'm pretty conservative, and it didn't offend me at all.
This is a great example of how powerful the use of such a word can be by a character who usually doesn't swear.
Only you can decide what sort of tone you're shooting for with your book, but unless you're writing for a specific demographic (MG or younger demographic, Christian/inspirational, sweet romance etc.), swears are hardly unusual or shocking in fantasy novels. Neither is the absence of strong swears, or even swears at all.There are some fantasy writers who don't have much, if any, swearing (Brandon Sanderson), or ones who have incorporated "made up" swears that lack the bite of real world swears for some readers and are preferable to others, and there are others who have quite a lot (Abercrombie, GRRM). Some are in between. I fdon't think I've seen F bombs in Robin Hobb's books, or even "shit," but I'm pretty sure there are some "damns" and milder swears.
Of course, who swears, how often, and in what circumstances will vary, just as in real life.
Some readers have strong opinions. There are some for whom even a single "Oh, God" is a deal breaker and others for whom the absence of any swearing feels too precious or prissy to be authentic. Nonetheless, fantasy novels that fit along the entire spectrum continue to be published. For me, it's really a matter of the voice and tone of the story. If I notice, "Hey, there's a lot of gratuitous swearing in this novel.
Everyone swears like salty the sailor in this world," or, "These characters seem unrealistically stilted or prissy. Soldiers who talk like visiting diplomats having tea with the First Lady on the White House Lawn?" then the author probably isn't doing a great job with the words matching the overall tone.