I had to skip the orc war drummers at first, as well (they couldn't hurt me, but I couldn't do damage faster than they could heal). Just after crossing into the farmlands, I met a travelling merchant (who you only see once, on the path to the cursed abbey) and bought a sword with better damage and a Stun bonus, and was able to go back and kill the drummers. When I replaced that sword, I learned the passive Stun skill.
Unfortunately, the difficulty does tend to get easier as the game progresses. I didn't start finding it too easy until the end section of the game, though, having maximized my level and reloaded (too much) to get good equipment.
