i'd say the theme is the underlying concept, the 'what i'm trying to say without saying it out loud' ideas. if you wanted to explore the relationship between an estranged father and son, that's the theme. doesn't matter if one's the court magician and the other is a jester (actually, that's a pretty good idea) or one's a klingon and the other is his bastard vulcan son. if you've got any drama at all you're apt to have themes even if you don't intentionally know you're writing them. for example, two sisters who don't get along might be a theme if it's something you delve into. you don't have to preach or teach anything to have a theme.
usually as i write, themes become apparent. it's my job to recognize them and see if i can exploit, er, explore it. that's another reason why outlines don't work for me personally. still, once i see a theme popping up, i take advantage of it and it helps shape the guts of the story.
if you use themes, you'll want to know about metaphors and allegory.
do you need a theme to get published? no, i wouldn't say so. would it hurt? probably not, unless you screw the theme up, and if you manage that the rest of the story is probably screwed up, too.
you can base novels around themes. were i to do this, i'd pick out a dynamic human trait, say, greed, and attach to that, say, redemption. or greed and and a riches-to-rags story (watch 'wall street' for the perfect example of this). basically, you can pick some dynamic trait and make it mean something.
just don't tell the reader what you mean. that beat up clunker your MC is driving could be allegorical for the way his life is going or the way society is progressing (or falling apart).
'good v. evil' doesn't make for a very good definition of a theme, imo. i mean, what is that? yeah, it's a theme, but could you be *less* specific? the answer is, of course, no, you couldn't. for it to be more effective it has to be more specific. 'g v. e' is, imo, simply too broad.
i'll give you an example. parke godwin did his version of 'beowulf,' which in its broadest sense could be a 'g v. e' tale. beowulf versus grendel. what the character is really about is his quest to forgive himself for being human, which leads him early in life to flee from a battle. he kept his action a secret and branded himself a coward, then to prove to himself he's not, goes on practically suicidal missions. after becoming reknowned for spectacular deeds and ridiculous bravery, he still can't forgive himself. he does eventually. then gets killt. such is life on paper, eh? the theme here is obviously redemption, and not just in other people's eyes, but in your own, the need to forgive yourself.
would that book exist without starting off without a theme? not without many, many drafts.
a story isn't restricted to only one theme, either. if your theme is the unattainability of the american dream, under that umbrella you're likely to explore many themes. in the 'beowulf' example, 'shame' would be another theme. he's shamed into needing redemption.
greed, power, friendship, lust, survival, jealousy, revenge, cowardice, love, growing up, loneliness... the list goes on. these are the things that give your story meaning. you have to test these things with your characters and situations. and you most certainly have, even if you know it or not. some academics would advise not to even start writing unless you know what you're writing *about*. looking at LOTR and harry potter, you can surely see many blatant themes being explored. the power of love and unity and friendship duh duh duh duh duh.
is it just about harry versus voldemort? hardly. is it good versus evil? well, yeah, kinda, sorta, in its broadest sense, i guess. it's kind of like saying 'conflict' is a theme. well, what about conflict?
*what about* conflict?
what about greed?
what about modern day slavery?
what about the street urchin living in the city of gold?
if you reduce your entire story into one paragraph, therein you're likely to discover the governing themes. you may have to read between the lines a bit, but it's there. 'a plucky street urchin befriends a slave and together they concoct a scheme to steal the street of gold.' right there you've got self-reliance/survival, friendship/potential love and a moral question: is it right to steal from the rich to help the poor (especially if the poor is yourself!)? my goodness gracious gravy, what more do you want from a story, anyway? sprinkles on top?
have i got you thinking about themes? good. and bad. it's easy to lose the story you're in the middle of when themes start to take over.
'oh, but, Preyer, not every story has a theme!'
name one. i don't care what the author's intention was, his story has a theme. whether or not he explored it is another matter.
'oh, but, Preyer, that's like saying every story has acts!'
true. what story doesn't? what writer doesn't start off with an introduction, rising tension, and resolution? again, you may not know you're doing it, but you're probably very much sticking to a loose storytelling structure. that's what people mean when they say 'i'm an intuitive storyteller.' what that really is saying is they're writing a story no differently than it's ever been written for thousands of years, but they lack the in-depth knowledge of what they're doing.
theme and structure go hand-in-hand. like peanut butter and jelly. fred and ethel. captain and tenille. preyer and hot, gratifying sex.
was 'the great gatsby' written without a theme in mind? i doubt it. 'friday the 13th'? probably not so much, but themes are there, you betcha.
take your character. he has to grow from point A, the first time we meet him or her or it, to point B, where we depart them after they've grown at the end. everything inbetween serves to justify that change. themes, when it comes to character growth, justify the justification. the character has to do or experience something for the change to happen.
'a man goes on a wild adventure and reconnects with his estranged father.'
pretty cool theme. you've got 'indiana jones and the last crusade' right there.
'a man goes on a wild adventure and learns that there's more to life than making money.'
hm, sounds like 'a christmas carole' to me. among other stories. in each example, you've got your beginning and your end *and* what your story is *about*. the 'about' is contained in all that middle stuff. it's how father and son build a relationship, how a miser comes to his life-altering conclusion.
you could do a story without considering the themes inherit in *any* story. you could also be missing out on a lot of great opportunities once those themes become apparent. then again you could always drown in your own self-righteous poignancy, you never know.
as to the comments regarding adding backstory... i'm of the mind that less is more and rarely, if ever, all at once and up front. dole out your world-building in spurts. if your 'universe' is an archetype in itself, i'll pick up on it pretty quickly. i don't need to know, nor do i care, about how the politics of frodo's village operate. i don't know much about sewage systems of the babylonians, either, and i rather doubt that if you told me it'd make much difference unless that's their escape route or entrance into the palace (and probably be very unimpressed with that old go-to cliche). do i need to know the history of how the city was established and every single conflict they had that's not germaine to the story?
if i don't need those things and the story works without that 'knowledge,' how is adding to it going to improve my reading experience? if you feel the need to add more details you think i'll find cool, great, pass the kool aid, but let's not overdo it, eh?
if you feel your story is flat, be sure that your hero isn't just being dragged along by the plot as opposed to actually doing things that will fulfill his quest, whatever that may be. your hero has got to make decisions for himself, he just can't constantly step into a tavern and run into the mysterious stranger with all the answers. he decides to do something and enters a new situation where we learn more about the world. and show what that world is.
biff decides to follow his destiny as 'the next in line as the chosen one,' and has to go to the armoury office to apply for an adventurer's license, which the bank requires in order to give him a loan to get started, as he can't even buy a sword without some coin. the armoury office makes him fill out form after form, then tells him he has to wait three weeks for 'processing.' in the meantime, he can obtain his adventurer's certification at the training grounds. the trainer asks about the whereabouts of his sword. biff explains that he doesn't have a sword to finish the training to get his certificate to get his adventurer's license to get a loan to buy a sword!
during this time, i imagine i'd be pretty well entertained, learn a lot about biff and about the kind of world he lives in. if, on the other hand, you want to toss in a few details about the luxury carriage trundling by, that's fine, too.
i could go on....