Takes nothing away? What if it isn't just wizard robes, what if it's the Robes of Vecna, should you be allowed to visually swap out those robes just because they "harsh your vibe" . What if you're given raiment from a divine avatar, and people comment about it when you wear it, but you're not really wearing it.
Should you be allowed to just willy-nilly decide what your character is wearing because you, outside of the game, want to play dress-up? I like playing dress-up too, but I don't want to do so at the game I'm playing's narrative expense.