The easiest way to break it all down is to look at bookstores. That's the reason we have these breakdowns, so readers know where to go to buy the types of books they like.
Genre fiction are stories that follow specific "formulas" or styles, and the novel depends on those tropes for the story to work. Science fiction & fantasy, horror, romance, mystery and western all are considered genre. There are subcatagories within these genres, like erotica (romance), cozy (mystery), or urban fantasy (fantasy), and books can cross over into mulitple genres. Paranormal mystery or fantasy romance. These are books found in their own sections in the bookstore.
Mainstream fiction is everything else. These are your stories that might have a splash of genre in them, but the bulk of the story doesn't rely on a trope for it to work. "The Lovely Bones" has a paranormal element, but it would never be shelved in the SFF section. These are books found in the "fiction" section of a bookstore.
Literary fiction is typically stories that rely on the inner journey of a character and not the external plot. The writing is more lyrical and the text itself is part of the appeal of the book. The prose is just as important as the story. These are also books found in the "fiction" section of the bookstore. Sometimes you see them in "literarture" if the store breaks them down further. Classics are also often found in the "literature" section.
Commercial fiction is anything that has wide commercial appeal, or will sell to a lot of people across boundries. Bestselling authors fall into this category. Stephen King is a horror writer, but you never see him the the SFF section because there are people who read King who would never call themselves SFF readers. Same with Michael Cricton. Janet Evanovich is a mystery writer, but her books are usually not shelved there, and she appeals to people who don't read mysteries. Commerial fiction is typically found in the "fiction" section, and often on special displays and racks that publishers pay for in the bookstore, because they are the first things people see when they walk in, hence they get noticed by the largest number of people.
As for submitting, think about where your novel would be shevled in a bookstore. If your novel falls into the tropes and format of a typical genre novel, it's that genre and submit it to agents who represent that genre. If your novel only has elements of a genre in it, but the story doesn't rely on those tropes, then submit to a mainstream or commerical agent.
DWSTXS, I'd call your novel mainstream. Depending on the writing, it could have literary appeal, but it sounds like a regular non-genre story.