The lack of proper resistance is the biggest issue. You don't have to alternate your playstyle. You just go "My physical damage are so overbloated it destroys EVERYTHING including ghost with its physical power"
Right now, the only counter to physical is dodge. And so far, only one fight has Evasive aura.
What I would have done is make Fin add 0.5 dodge rate and Wit increase accuracy. Con adds a 0.5% physical damage resist each point. And make magic wand hits anyway.
This way, magic, while delaing less damage, will be the more reliable option without the need for Wit. Con make it possible to reduce some physical damage. And that Wit will be needed in all physical build to keep hitting instead of going 40 str/fin asap.