In general, fiction is divided into literary fiction and commercial fiction (also called mainstream fiction). There aren’t any hard and fast definitions for one or the other, but there are some basic differences, and those differences affect how the book is read, packaged, and marketed.
Literary fiction is usually more concerned with style and characterization than commercial fiction. Literary fiction is also usually paced more slowly than commercial fiction. Literary fiction usually centers around a timeless, complex theme, and rarely has a pat (or happy) ending. Good examples of literary fiction are books by Toni Morrison and John Updike.
Commercial fiction, on the other hand, is faster paced, with a stronger plot line (more events, higher stakes, more dangerous situations). Characterization is generally not as central to the story. The theme is very obvious, and the language not as complex.
The biggest difference between literary and commercial fiction is that editors expect to make a substantial profit from selling a commercial book, but not necessarily from selling literary fiction. Audiences for commercial fiction are larger than those for literary fiction. Tom Clancy, John Grisham, and Stephen King are all prime examples of commercial fiction authors.
One of the easiest ways to determine whether your work is literary or commercial is to ask yourself, “Is my book more likely to be read in college English classes, or in the grocery checkout line?”