The only ones that have ever managed to be profitable are the Bethesda Elder Scrolls and Fallout games, and frankly they're far weaker examples of what potential the genre has to offer than System Shock/Ultima Underworld games. I think this is because the Bethesda ones are dumbed down in what you can do, plus they hold your hand with missions and always tell you what to do and where to go and mark stuff on a map, which to me detracts immensely from gameplay. I grew up playing games that didn't tell me where to go or what to do aside from hints, and expected me to use my head to figure stuff out for myself.

I think many modern gamers are incapable of this, or find it too much of a challenge.

I found combat the least of what makes these games interesting. If I want combat, I have doom. The exploration, puzzles, immersive story and general figuring out of "wtf is going on" is what makes this genre great for me.