This has been an issue with this system since Divinity Original Sin 1. It's definitely something they need to work on.