I agree that there should be a penalty for swapping anything from your inventory while in combat. It seems that should cost an action, or at least a bonus action. But I can't see that for switching between equipped melee and ranged weapons. Even on our avatars, they show the other set on our backs, in easy reach. And for some classes it's common to have to switch between them in combat if there are no enemies you can get in melee range for on that turn.
But where would it make sense to put the torch other than your inventory? Hanging at your waist? Especially since one of the unrealistic things about these torches is that they stay lit in your inventory, but they don't burn you or anything else you're carrying!
Actually, I like the idea of a separate torch slot. Once you put a lighted torch into it, it acts like the melee/ranged buttons. Out of combat when you click on any of those buttons - or click a melee/ranged attack - it swaps the torch in and out of your inventory for you. But if you find yourself in combat with the torch equipped, then once you select a melee or ranged button/attack, it automatically drops the torch on the ground as you bring up your weapon(s). Because why would you take the time to put their torch away in their pack while in combat? That would make sense at least.
And if you end up dropping it in water, that's just the cost of doing business. After combat you can pick it up and relight it somewhere. Can you light a torch from a party member's torch? I never tried that.