Summons are ridiculous because there is no chance to fail. Every one of my characters has a way to summon. My Wizard has everything wizards can summon. My Tank has offspec Pyro so can summon a Fire Elemental. My 2Hand damage dealer has offspec Witchcraft, so can summon Undead Warriors. My Marksman has offspec Scoundrel, so can summon a Wind Up Toy. The fact that these can be out PLUS I can have something charmed completely refocuses AI attention. I don't know why their priority is so high. For instance, there is a fight near 2 trap doors with a Pyromancer and 2 or 3 fire elementals. Summoning a Fire Elemental, they ALL focus on it, and because it's fire-based, he survives and tanks ALL 5 turns before they even start on my characters. Maybe their resistance needs to be not 100% as well.
Man-At-Arms skills are pretty balanced for the most part. Flurry is a skill that hits about 4 times but has a long cooldown. Dust Devil/Whirlwind also have a long enough cooldown. Battering Ram as well, and its chance to knockdown isn't that high and it doesn't do a lot of damage. I listed how I thought they should change in general. Mixing up damage types may be a good idea so some are useless vs. certain enemies (ie. Undead).
Generally the Man-At-Arms skills should be pretty high damage considering you're in the fray of battle. They're all pretty useful, but none too OP (save for Flurry maybe at times). I'll go through some more to see if I think they need changing.
Thanks for trying to mod the game for a harder difficulty!

-Edit-
My thoughts on summons are above, though suggested skills are just off the top of my head. I think they should be more about utility and less about damage. Make them useful in certain situations.
-Fire Elemental can be about damage and path blocking. Firefly, Smokescreen, Small Fireball, and/or even Phoenix Dive or Explode/Self-Immolation. Fire Rain would be nice :P.
-Ice Elemental can be about healing. A couple healing spells and Rain. MAYBE even ice Wall, but that's pretty strong and I don't know how strong it would be from the elemenetal.
-Air Elemental can be about delay tactics. Make Invisible, Shocking Touch, Teleport.
-Earth Elemental can be about damage prevention/buffs. Blessed Earth, Earth Resistance Shield, Fortify.
As for other summons...
-Spider should be about slow and poison. Acid Breath, Midnight Oil, Magical Poison Dart.
-The Bloodswarm can be really annoying bugs. Infect, Nature's Curse, Mass Weakness, maybe give THEM Firefly instead since it's funny.
-Wolf should be a howling inspiration. Inspire, Encourage, Lower Resistances, maybe even Rage for itself.
-Undead Warrior can be a weak warrior. Draw Blood, Dust Devil.
-Armored Undead Warrior can be a better warrior. Battering Ram, Whirlwind, Flurry.
Also consider these summons start with a ton of AP, meaning they might be able to use 2 spells in one turn. With that in mind, they really shouldn't be allowed out that long IF they have such amazing utility. It should be like...2-3 turns maybe. If you go with a simpler approach, more turns is better I guess. Or if they're weak. If their fireball only hits for 100 at level 20, it's fine. Same with Ice Wall and such. I just don't know how it'll scale =/.