What's the point of silver arrows if all special arrows ignore armor?
Two things:
1. You are not ALWAYS shooting special arrows. So you will do a lot more damage if you shoot a silver arrow at the beginning of a round, and then proceed to use normal attacks on that opponent for the next two turns than you would if you just used all normal attacks.
2. You are not necessarily the only one attacking someone with physical attacks. You likely have a melee attacker or two in your party. They'll also do lots more damage for the next two turns if you use a silver arrow.