The Barbarian feature could use a better description that clarifies that is sets your character's BASE AC to a new value.
I would normally agree with you, but it's not a Barbarian exclusive problem. In 5e, AC calculation is explained and tells you how things interact. You always take the highest value of AC amongst features and spells, certain items and spells can raise AC like normal, some can't, things like that. So honestly, not clearly explaining AC is the real problem.
And really, Unarmored Defense isn't changing your base AC. Base AC has always been 10+DEX modifier. The feature is treated the same as Armor calculation, but doesn't require you to use armor.
