Honestly I do not think that allowing characters who can cast spells to cast spells without needing to perform the cumbersome actions of putting away or dropping their weapons/shield every time is a game breaker. If you leave it in, people will complain about excessively long animations, and they wouldn't be wrong.
D&D is designed to be flexible and the DM can put in house rules for all sorts of things to suit the tastes of that particular table.
