Magic has several handicaps, the worst probably that it is much harder to boost your magic damage. For physical you mainly need a better weapon/shield for a huge boost, magic damage mostly relies on level.
Magic might have a lot of damage, but enemies are mostly spread out widely, so you need to prepare first to really use AoE effectively. It ist much easier to just focus the enemies down one by one and put them into a 'doom zone', where you will knock them down every turn until they are dead.
And yes, every fight pretty much feels the same, which is why my friend and I stopped playing in Act 2.
