Shields increase your AC even when they're not equipped. You can use a Heavy Crossbow while getting that fat shield AC bonus. Remove Shield AC bonus if the shield is not being used.
That means the free equipment switching also needs to go. Otherwise you can just cheese end your turn with a shield equipped regardless of what you did or had equipped.
Stuff like this add tactical depth. When you can't just do everything and have to make choices, the game becomes harder in a more meaningful way than just giving all enemies some arbitrary stat bloat.
