i think that physical damage is probably stronger overall due to the omnipresence of high resistances to some (or sometimes all) types of magic
i don't think anyone should reasonably call physical damage dealers "easy mode" because of this though. the disparity is really not that big and magic users have access to stuff like summoning and necromancy for bosses with high magic resistances. even if one is a little stronger than the other, there are ways to play either (quite a few of them in fact) that allow you to play through the game on tactician without too much trouble
there's no need for this kind of pissing contest