This person says he played 30h with his wife and had no problems. Maybe there is something we do other than getting more spells/items that makes it degrade in time like, getting more origin companions or something.
Would try to play while only recruiting 2 for example but i can't be bothered given how Larian has been silent so far anyway.