Make sure he's not wearing any gloves, boots, or helmet that count as armor. Clothing should allow for the Unarmored Defense boost.

Mage Armor does not stack with Unarmored Defense.