Probably depends a lot on your party make-up. For example if you had a geo mage you'd probably get a lot more use out of fire arrows.
Do you mean the high bb/wp means charm arrows have a low hit rate, or that the enemies are higher level than you (do special arrows not scale with dex? I play coop with a friend and he is the ranger)?