I think WoW had a reasonable answer to the semi-restricted transmogs, which was to restrict them via armor type. Like, Light Armor could be swapped with any Light Armor you've unlocked, but you couldn't swap Light Armor with Heavy Armor.
Though, personally, I'd love complete freedom in single-player, and perhaps give a multi-player host the option to set "Unlimited Transmogs" vs "Limited Transmogs" vs "No Transmogs," to control the level of fidelity to what's seen on the character model. I adored some of the non-armor (and yet non-camp clothes) items but could never wear them, and even within my armor type, I sometimes liked the look of pieces that just didn't have the stats that were right for me.
