The biggest downside is the movement speed, but you could use cup of oil on all armor pieces so you are wearing heavy + move speed, or, I guess you could just say eff it and wear medium. In fact, I have tenebrium 7 on my rogue, I could just slap a two-hander on her, buy whirlwind and dust devil and battering ram and try it myself with a tenebrium 2 hander.