To be fair, I believe Silence is one of those cases where the rule can't be incorporated in it's full capacity because of technical limitations. Now I haven't played every d&d crpg out there but I've never seen the spell being able to affect the environment before, i.e silencing the drum or the area around the wall.
"For the duration, no sound can be created within or pass through a 20-foot-radius sphere centered on a point you choose within range. Any creature or object entirely inside the sphere is immune to thunder damage, and creatures are deafened while entirely inside it. Casting a spell that includes a verbal component is impossible there."
Now, for this to work as intended, Larian would have to predict every single scenario where a player might see use of casting silence on the environment, since there won't be a DM able to make an active decision. Or completely recode how sounds work in the game. And that's a lot of work either way.
But I do agree it should completely be able to shut down any action that logically needs a verbal component. For example by affecting a creature's vocal cords. And that can't be that hard to code in. Just make a hidden checkbox for spells and actions that will be able to be affected by the spell.
Characters waking up because of sound, calling others, etc already has to be programmed. They just need to add a check for silence to the conditions to see if works or not, which they clearly haven't done.
It might be a long progress and could cause bugs, but that's another story.
