Torches are too powerful as weapons (2d4) and too complicated to light. I mean you can light candles and torches on the wall with a single click, but not your own.
Holding a torch offhand for non-darkvision characters to get rid of disadvantage in combat sounds about right. Or throwing them on the ground to have a small light radius while using a shield.
