Yeah the no free spell hand rule is going a bit far. That and restrictions around drawing weapons are probably the most hand-waved rules in 5E. No one wants to manage that crap. It’s not OP just to let that go. Dungeon Dudes did a good video about it recently, worth a watch.