A commented asked...
"I would like to see you write about novels that are so original and different they fall outside the desired mainstream. I suspect there are a number of us who fear we just may have written this type. I would also like to see you address the issue of whether an agent will be interested in novels like this at all. Is this a non-starter?"
I'm going to preface this by saying I'm not an agent or or editor, so everything I'm about to say is purely my opinion based on what I've read and conversations with those in the biz.
Publishing is a business, and there are well-established genres to help readers find the kinds of books they want to read. Bookstores didn't just arrange their shelves that way because they look good. It's all about getting books into the hands of customers. If your book falls outside one of these categories, it can indeed be hard to place, let alone know who to submit it to.
Let's tackle the first part of the question... what to do if you've written something so original and different it falls outside the desired mainstream.
At the risk of sounding harsh (you know my goal here is to help, not trash, writers), the odds of writing something honestly unique are slim. Just about everything has been done before. The upside to this, is that means you probably don't have to worry about it. Odds are your work is either "fiction" or "literary" and you can market it to agents and editors who represent these categories. Lots of different stuff is shelved under fiction.
On the off chance you really have written something unique, well, I think it all depends on how unique. If it doesn't fit even a little with anything else out there in the marketplace, not even general fiction or literary, odds are there's not a market for it. If someone were buying that kind of book, you'd see others like it. If readers aren't buying it, then odds are editors aren't either.
But that doesn't mean the book is doomed. New genres and subgenres appear every year. Different types of books get published and shake everyone up because no one saw them coming. Every genre had a first book to try that genre.
To quote one of my favorite disaster moves, "There's no history of anything until there is."
Now, for the second part... would an agent be interested in this type of novel?
I think agents are interested in great books they can sell. If something unusual crosses their desk and they love it, and they think they can find at least one editor who will buy it, they'll probably take it on. But loving it isn't enough. You hear about agents who passed on a book they loved because they didn't think they could sell it. Some of those went on to find homes (and sales) elsewhere, others never did. Selling it trumps loving it.
I do think it's possible to write something that is so different and out there that there's no market for it, and therefore it won't sell. If you have this type of book, you might indeed be out of luck. It stinks, but that's the business. Plenty of great products have failed because they were either ahead of their time, or just too different for the folks they were selling it to. Not just books, either.
If you think you have this kind of novel, you have three options:
1. Give up and either start something new or self publish. (yeah, I don't like this advice either, but I had to say it)
2. Find things about the work that are similar in some way to what's out there and market it toward agents who rep that kind of work.
3. Revise to make it a bit more marketable and submit to those who rep that kind of book.
In most cases, the odds are you have something that is unusual, but not unique. It doesn't have to fit a genre, and I see writers all the time who say they can't place their novel because they're trying to force it into a genre and it just won't go. Fiction is a perfectly acceptable genre and it allows for all kinds of books. If you don't know where a book goes try that and see what happens.
Something about even an unusual book is likely going to be comparable to something else, be it the style, the tone, the setting, the plot, the premise. Even if it's small, you should be able to find something that makes it kinda like something else. At least enough to pick agents to send it to. Think outside the box.
The "desired mainstream" is just what's selling right now. It changes on a whim, and you can't predict what will be popular. Harry Potter's first print run was 1000 books. JKR sells more than that a day now. The Shack was self published as a Christmas gift and then went on to bestseller status. The only thing you can count on is that readers want great stories. And what they consider great varies. A lot.
Write the story that you feel the most passionate about. Market it as best you can and submit it to those you think have the best chance of buying or repping it. That's all any of us can do, and as long as we keep trying, we increase our chances of success.
I know that sounds cheesy, but it's the truth. A "too original for mainstream" now book from a debut author might be a breakout blockbuster from that same author after they sell a few more books and build a following.