"For now the game mostly penalize you for not playing it in multiplayer."

Given how the four available companions fill all the rolls needed... (except maybe a dedicated fire and earth mage)

Not seeing the penalty.