I dunno ... it depends ...
This idea can easily become something cool, aswell as something players will hate with burning pasion.
It takes me back to Dragon Age: Inquisition ...
Personaly i hated those slippers we were wearing at camp. :-/
Especialy for mage characters ... i get that Warrior wanted to get rid of that armor for a while ... even if some characters (Blackwall) were simply wearing different kind of armor instead. :-/ But why mages? They are allready wearing simple light robe ... no armor.

//edit:
And you should not forget that you are sleeping in wild, with litteraly no protection around. :-/
It seems lika good idea to sacrifice comfort and keep your armor to get some protection.