I've not played physical damage dealers enough to know whether they need nerfing or magic just needs buffing/reimagining. Both should be balanced against the encounters in the game and offer roughly equal challenge.
If summons are a factor in balancing casters, their damage certainly ought to scale with int. As it is, there's no real barrier to summoning by non-mages.