I found summons just as useful in the late game as early on, if only to tank a few hits. They aren't so bad when there's just one, but when three or four of your characters, or even just two, can summon with barely any cooldown, they make fights a lot easier without any effort. Enemies love targeting summons especially since you can plop them down right in front of them. Even if a summon eats only two highly damaging hits, that's a lot of damage your characters aren't taking.
My solutions would be:
Give summons much longer cooldowns so that even with high int, it takes 5 or 6 turns to summon one again.
You need the proper elemental surface to summon. Perhaps the surface is consumed when you summon from it.
Summons have less health and the Stench talent so enemies target them less. Enemies should no longer target elementals that they heal with their attacks.
Summons get two or three utility spells. Maybe stuff like cleansing water, cleansing fire, feather fall, perhaps even something like battering ram for an earth elemental.
More enemies (but not too many) get the dispel summon spell, and also charm.
The Source Difficulty mod did a lot of this already and from what I've seen with that mod they worked pretty well. These changes might sound harsh, but summons are pretty absurd at this point. Requiring an elemental surface would in particular address a party having up to four summons, because it would be difficult to get enough elemental surfaces to get four summons unless each party member had different elementals at their disposal and they spent time setting up the battlefield. But if someone wants to play like that, be my guest. A summoner party shouldn't be invalid, it just shouldn't be quite as effective or as easy to pull off.
