Activating a mod clearly seems to take a long time. It doesn't seem to matter what the mod does or how big it is. Definitely seems to be an issue that Larian has to solve, not modders. I assume the save has to be updated, perhaps with entities throughout the game needing to be updated. It's pretty amazing how long it can take to "update" though, in the order of 10-15 minutes. It's also unclear if activating multiple mods at the same time makes a difference.

In general I've found my load times to be perfectly fine after activating mods though. So no idea what's going on.