something tells me this shouldn't be allowed
Yes, there are definitely things that Larian needs to restrict in here. :P I'm sure they are working on it.
Well, I'm really not perturbed by the possibility of this weapon-swapping exploit. Like I'm not bothered by barrelmancy in the slightest.
Because it is far from being trivially accessible, and in fact requires a certain effort to execute. And it does't feel like the game makes it mandatory to use. So I feel it's one of those few cases where "if you don't like it, don't use" actually applies well. This is very different from Backstab, which is one click away, or High-Ground, which forces your hand because enemies will run for it if you don't occupy it.
Larian
loves giving players ways to abuse the game. Some players are bound to like it too. At best, they could implement an option that allows players to customise their experience and disable equipment change in combat (as well as inventory swaps, etc).
But I'd rather have Larian focus on more fundamental issues first (like the controls, among many), and keep this kind of customisation option for the end of Early Access.