Short answer: No.
Long answer. Of course not.
Charisma is not looks. If you think a low charisma character can't be physically attractive you are simply wrong. Check the rulebook again.
If you want to homebrew this then explain why good looks affect intimidation? Or spellcasting?
