E.M. Forster defined the novel as "a work of fiction of over 50,000 words." That covers an awful lot of ground, so it's helpful to classify novels further. Here's how editors and publishers usually make those classifications.
The Genre, or Category, Novel
In their settings, plots, and other devices, these books are immediately recognizable types. They include westerns, romances, mysteries, science fiction & fantasy, horror, and action-adventure.
Usually, these novels are published only in paperback. Often, publishers have certain criteria they want their category novelists to meet, including a specified word count, the amount of sex or violence, and/or the moral tone. Category novels usually have a shorter shelf life than books originally published in hardcover and command smaller advances as well.
The Mainstream Novel
The mainstream novel is the broadest classification. Mainstream novels are usually published first in hardcover and sometimes reprinted in paperback or trade paperback (a larger and sturdier version of the mass market paperback). The plot of a mainstream novel can concern anything from the story of people trapped in a mineshaft awaiting rescue, to a woman's struggle to succeed as a clothing designer, to the story of a young boy growing up on a farm in upstate New York.
The plot and setting of a mainstream novel may determine that it also fits into one of the categories just described: Danielle Steel writes mainstream romances; Stephen King writes mainstream horror; Sue Grafton writes mainstream mysteries. However, mainstream genre novels are generally longer, less predictable, and peopled by more complex characters than their category counterparts. A mainstream novel also aspires to shed some light on society and/or human nature, while genre novels need only succeed as entertainment.
The Literary Novel
A novel merits this coveted label when the author demonstrates a masterful command of language, along with true depth of insight into character and society. Literary novels sometimes have tight, suspenseful plots-and a tight, suspenseful plot can only be a good thing. But the plot of a literary novel will ultimately be subordinate to the author's study of the characters. And, while category novels and many mainstreams are set in exotic locales, against the background of glamorous industries such as film or fashion, literary novels are just as likely to deal with everyday life in ordinary settings.
A literary novel may also be mainstream if it is accessible enough to garner a wide audience. We might say that the literary mainstream is the Holy Grail of novel writing, since it potentially brings prestige, celebrity and financial reward to the talented authors who write them. Literary mainstream writers include Amy Tan, Barbara Kingsolver, and John Irving. In their day, Dickens, Hemingway, and John Steinbeck achieved this pinnacle.
Literary novels are also usually published in hardcover, but they are especially good candidates for trade paperback. A trade paperback is less expensive to produce, and so publishers see it as a natural format in which to launch a literary novel, which may have a smaller audience.