Short answer: No.
Longer answer: Not at all.
Longer still answer: Damage is based largely on your equipped weapon. Without equipping a weapon, your damage sucks. Your damage already falls off terribly if your weapon is a level lower than the enemies you are facing. You'll never keep up in terms of damage if you are not using a weapon.
