I guess you're taking unarmoured defense for style? Otherwise save yourself the level and maybe get to a feat with rogue (dual wielder might suit the flavour you're going for while providing an actual benefit).
Don't see any reason this should get smashed in fights though. The only obvious weakness is trying to make unarmoured defense work - you'll incur a high opportunity cost to get up to the AC you could otherwise get by just chucking on medium armour.
