Right, I would say swapping between weapon sets should be free, but if you want to equip another weapon to replace one of the sets, then that should cost AP. It really ended up punishing dual wielders the most due to having to equip two weapons.
I could see weapon swap opening up some new and interesting possibilities... Like maybe you have two crossbows and can fire twice on your first turn with swapping.
