What difficulty are you playing?
On Tactician there are plenty of opponents with evasive aura (which they don't have on Classic). This makes a pure physical more problematic.
You also have the demons in hall of Echos - one 100% immune to all magic and one 99% immune to physical.
Luckily for me (solo scoundrel) the skill Sawtooth Knife (unlike all the others) doesn't count as physical when it comes to resistances - it directly hits vitality even though the tooltip says piercing. That surely must be a bug assuming piercing damage is physical.
I play on tactician only. There are only a few enemies in the game with evasive aura, which is easily negated by stuff like glitterdust (if you actually equipped it) or medusa head (pertify nulifies dodge). Like up to the end of act 2 I only recall one(some black ring dude on advocate's island) having it and tbh I don't remember enemy stats very well after that cause everything just explodes at that point due to the silly scaling, but maybe there were a few more. There are enemies with elemental resistances in almost every single encounter, often pretty high resistances across the board. I think the game would be a lot more interesting with much higher number of high dodge enemies (not 90% but more like some common enemies having 25-50% dodge). Would actually give accuracy boosting abilities and runes some use.