I think that the complex genre of "literary thriller" is often misappropriated, but if you're searching for a perfect example of a book in this category, you need look no further than Leonard Chang's 2009 novel Crossings.
Why is this book literary? Because the stark language is mesmerizing in its simplicity and its power. Because the characters are difficult, complicated, and heartbreaking. Because the reader is transported to a world that is terrifying and tender, a Pandora's box wrenched open in the first chapter and taking the entire novel to (possibly....) be slammed shut. Because cliches are absent and rawness abundant, because the story is more important than the plot.
Why is it a thriller? Because I couldn't put it down. I read the second half in one setting, ripping through the pages, because I HAD to know what was going to happen to the characters. I became so caught up in the fate of the characters that it was all I could do to not skip ahead so I could find out, for God's sake, how the characters were going to untangle themselves from the hopeless web they were all caught up in.
It's been a while since I've read a book that I felt had momentum, driving me forward to read just one more chapter (!), and I was greatly appreciative of Chang's ability to lure me in with his literary treatment of characters and then keep me reading with his fast-paced and twisted plot. Crossings is a true literary thriller, and, at the end of the day, simply one hell of a good book.