If you are trying to break the game, then don't complain when it's broken.

That said, I do think summoner is a problematic skill because it's all or nothing, because it interacts with basically nothing else. The only way to buff is to buff summoner level, and when you do, you end up with crazy powerful summons.

It seems to me that summoner should be a support skill set. Make the fire slug, oily blobs, etc. scale with pyro and geo abilities and summoner ability -- and make all summons spells require points in summoner. Thus, it will be helpful for elemental mages to get summoner points if they want to use those spells.

Meanwhile, summoner can focus on things like totems and traps, both of which should interact with attributes like INT or STR.