As stated, the only 'downside' would be the movement reduction compared to robes.
But then you have to wonder yourself... since when do wizards need to walk much anyway?

So yeah, perfectly viable...