That is correct.
Anything you add to a character only affects that character, unless specifically noted such as Leadership affecting everyone ELSE in the party that can see the character.
Thus, if you increase Perception on a character, THAT needs to be the character you walk around with while the others follow. If you add Charisma and/or Politician to a character, THAT needs to be the character doing the talking with NPCs.