Shields are the actual problem here. They should cost an action to equip or unequip instead of having zero drawbacks and an exploit to get that +2 AC even if you're using a Heavy Crossbow or Greataxe.
+1. RAW, Shields cost an action to equip/de-equip.
Hmm, I do think it's a too easy to swap weapons, but a full action seems harsh, basically you'd miss a whole turn if you want to swap weapons unless you were hasted or could attack with your offhand. [...]
I'd be okay with something like giving characters the option to define a set number of weapon sets (3 or 4) and giving them the ability to swap freely between them once per turn, but they'd need to use an action if they wanted to swap again or equip a new weapon from the inventory or the ground (which latter I think is the case now).
This is the way. With the ability to swap freely back and forth between your weapon sets until you actually attack with one of them. Then, if you already made a swap this turn, you can no longer swap freely.
