It’s sometimes useful, when you’re starting to write and define yourself as a writer, to think of broad categories of fiction, because it can help you understand what the reader expectations are (and decide whether you want to deliver them or subvert them!) If this kind of thinking is not useful for you, I absolutely do not believe you have to think about it at all — write what you want to write, and let the market and your agent figure out where and how to sell it!
But for me, as part of the study of fiction, it’s been instructive to look at the differences and similarities between different categories or markets. One of broadest is genre fiction and literary fiction.
I think the easiest and incidentally the most useful way to think about genre fiction is, it is writing that intentionally serves the expectations of a certain genre. Those expectations can be plot-centric, as in murder mysteries where we expect that the murderer will be revealed by the end, or emotional, as in a thriller when we expect to be thrilled, or both, as in a romance, where we expect the emotional fulfillment of a happily-ever-after ending that also ties up the love plot.
While literary fiction can have any or all of these elements, it also tends to be harder to categorize in a particular genre; it is also, quite often, more invested in exploring theme or as aspect of narrative technique rather than focusing on plot or emotional impact.
The main thing to remember about such categories is they are not prescriptive; at least, I don’t think it’s fruitful to treat them in any way as prescriptive. There’s no real formula for “mystery” other than the genre expectations, and even that can be broken by a skilled and confident writer, to good effect (that one mystery story where the narrator turns out to be the murderer, for example!) Instead, I use categories to help me think about my story after I’ve written a draft. It can help me focus on what I realize is most important, or it can help me understand a potential reader for the story, and what they’d expect. For me, the most helpful way to frame genre questions has been to ask myself, what other books or texts is my story in conversation with?
What’s interesting to study is when novels work in both categories. Erin Morgenstern’s novel The Night Circus is a genre fantasy romance novel about young star-crossed magicians. It is also a literary novel of surrealism exploring the power of dreams and imagination, with themes of family legacy and professional rivalry that threaten to destroy innocent lives.
Another example is the short stories of Kelly Link, who is a spectacularly gifted and literary writer whose dazzling style is as much a hallmark of her work as is her uniquely fabulist, surreal imagination. See also Carmen Maria Machado, a fantastic writer who’s published in both literary magazines like Granta as well as genre publications like Nightmare. She’s also written a memoir, In the Dream House. So don’t ever feel like this genre exploration needs to lock you in to one field or another!
One way to start understanding your own work is to read what other people write. Read a lot, read across different genres. I learned so much about emotional scene-building from reading romances, for example, and so much about plotting from reading mysteries. I learned a lot about subtlety and letting small details do the heavy lifting from reading literary fiction.
And I learned that ultimately there’s really very little difference between these categories, at least in terms of the broad outlines. I found that very comforting — it freed me up to write what I wanted and trust that there will be readers out there for it, regardless of what genre my story ends up in!