Most physical skills get their damage from weapons, so if you don't have any, the damage will me much lower.