It’s a pretty nice idea, and might not even be too hard to balance. Perhaps the game drops the difficulty down a notch for any fight against a human opponent.

The main potential problem I see is what the enemy player would do during all the non combat parts. Would they just watch the party player(s) as they explore the world, talk to NPCs, do some shopping, try to chat up companions, etc for an hour before they get to participate in a 5-10 minute fight?

I think they’d need to add some extra features to make this more interesting for the enemy player or they’ll get bored quickly. I’ve no idea what they could be though.