It is in the description of the Charm spell. It is a combat spell, so of course people will get mad, if you use it on them.
It isn't even important, if you convince that guy to join the party or not. If you fail, he will just say "No, thanks" and nothing happens.
The spell works as intended, so nothing for Larian to change here. But maybe wait with HM, until you have a grasp on those spells. That is not meant mean spirited, but spells like friends and charm person can ruin your HM, if you don't know about the consequences.
