“I’m not sure what you mean by this, but something is either commercially viable, or no publisher will want it. Even small, literary publishers have to make a profit or go out of business.”
I mean that if you (generic, we’re assuming unpublished writer) write, say, a thriller that doesn’t resemble another successful novel, you’ll scare publishers. It really doesn’t matter if readers might enjoy it. The publisher won’t know that. He only knows what has done well, not what (original work) will do well. If the publisher can’t find a semi-match (“Like Blood on the Chandelier, but featuring a midget instead of a sword-swallower”), it spells risk. More risk than the novel that’s similar to a bestseller and therefore has a proven market. Assuming it isn’t garbage. Maybe even then.
Harry Potter was big, and now they’re spewing out middle grade novels that feature portals to magic worlds. Cornfields, books, secret doors. No risk as long as the kids can’t get enough magic fantasy.
An established and successful writer reduces risk to the publisher, and so can be more original. Except that he or she runs the risk of losing the following that’s been built up. It’s a jungle out there.
If you could go back in time to the sixties and seventies and look at library shelves, you’d see lots of novels that didn’t fit any genre, and yet weren’t literary. Usually focused on odd characters and/or odd situations. They were probably chosen because they were unique. No suits to keep the foolish editors in line. Risky business. Inefficient business, too, but it had its charm.
Since the writing is valued much more in a literary novel than in a commercial novel, I think a literary writer can be more original in terms of story as long as she meets the prose standard. For example, a literary novel set in Bulgaria, with Bulgarian characters, is not taboo. When was the last time you read a commercial novel by an unknown that incorporated that sort of foreign setting/foreign protagonist? Outside of historical novels.
Before publishers became Truly Corporate, they weren’t $mart enough to cut out all possible risk. Well, they still take risks by paying huge advances to big names that flop. Or fail to predict the sudden end to a subgenre’s popularity. No one’s perfect.
Time to hit the sack.