The Weapon set / dual wield (V) toggle works strangely. I found it just switched me into using either melee or bow.
The only times I've seen the Off hand attack, is when you manually use it (Right side toolbar) as it costs a Bonus Action.