i was having this same issue with some friends. upon deleting the entire \steamapps\common\Divinity Original Sin 2\DefEd\Data\Mods folder we were able to connect to each other without issue. this of course also fully removes the party size evolved mod from the game
