There should definitely be consequences to casting in front of people. No one wants Charm Person cast on them, and it's half of the reason why sorcerer's Subtle metamagic exists.