Take a look at the combat log. It should tell you the saving throw for the enemy. Also, if the enemy is immune it will come up with immune to charm, so that isn't the issue.
If special arrow hit chance doesn't scale then base chance is always 100%. If you're higher level than the opponent it should go up 10% per level and the reverse if you're lower level. Additionally, each bodybuilding or willpower that the opponent has will reduce the hit chance by 10%. So the problem may be that later in the game enemies' higher bb/wp will reduce the chance of hits.
If special arrows don't scale, rapture most certainly has a higher charm chance (goes up 5% extra for each dex point above the base dex requirement).
Personally I think special arrow hit chance should stay pretty much the same throughout the game, that would be nice.
Also, I'm sure silver arrow > one autoattack to set up the rest of your team will be better than 3 autoattacks vs tough enemies. Is there a disease arrow? Can't remember seeing one...
