After getting the battle tower you can visit the trainer to increase the level cap of your preferred weapon specialty or favourite combat skills.
Before getting the battle tower I was using two axes. One of the first places I went afterwards was the sinister cave in the fjords, and on finishing that I started using Barnabus' hammer, which was a significant upgrade from my main axe. I found another hammer shortly after that which was almost as good, which overall gave me a pretty good damage boost fairly quickly.
Charms and enchantments can also make a difference, even with the same weapon(s).
The type and level of your opponent comes into play for damage calculations.
Character Optimization
