I think the game will work out this nuance at one point. Currently it only silences casters that require verbal components. It should indeed be relevant in other situations.