I honestly think that is how CRPGs are always going to be in terms of popularity. How popular they are is going to go in waves and I think Obsidian left the space after Deadfire because:
1. They botched the launch. I didn't play at launch but I heard it was bad. Really bad. I have played the game since they fixed it up and I honestly think it is great.
2. Player burnout.
1. The game underperformed and figuring why isn't
an easy thing2. After working on two cRPGs + DLCs back to back, team burnout happened. With Deadfire struggling to make it's money back, there was little reason to push for a sequel and not do something fresh.
2c from me:
Yes, the launch had issues (mainly mid-late game balance, and dodgy reactivity/reputation gain), and the game was and still is flawed - still any problems Deadfire has or had would be discovered by people who played the game only. I still believe that Deadfire sales are result of PoE1 - if vast amount of player base wanted more more of PoE after PoE1, they would have come for more.
Edit:
Rearching through Josh's Tumblr I found this
little post. Turns out Deadfire made some profit in the long run, though it took its sweet time.