Well, one issue is that you'll miss Charisma opportunities (never knowing they exist) unless you keep Charisma up most of the time. On the other hand, it's only 1AP to equip rings or amulets during battle. Keep them in a pouch. Alternatively, experience and scouting will eventually tell you which situations don't benefit from Charisma, but I try not to savescum too much.
