Stop wearing armor!
Be Naked!
So sayeth the wise alaundo.
Serious answer:
*I think* - because the norm is Spellcaster doesn't meant to soak in damage. You meant to outwit it. Just like Rogue, you're not meant to Tank.
Of course depending on what's given to you (item) in the game, you can do out of the ordinary thing, and you can do it in this game, but with caveat.
Since basic caster doesn't have Medium armor proficiency maybe you need to multiclass to fighter or warlock or cleric for their armor proficiency.
ps.
It's not silly for robes to have high AC, especially when it's infused in Magic!
