A 3-10 range of cooldown times does not make me like the idea any more than a 1-7 turn range of cooldown times. That skill has to be powerful enough to be worth whatever cooldown time it has.
In the EE, that is determined essentially by the power it gets from the associated primary attribute. The higher your INT the more damage it does and/or the higher chance it has to land a status.
You want to add in complicating factors of hefty penalties for not optimizing your build for skill X. That highly promotes min-maxing specialization and kills off any use for hybrid classes. I don't see where you get the the idea that punishing not being a pure specialist opens the doors to new builds.
What happened to this:
All that make a sense.
If something works, lets rebuild it from scratch.
Oh, wait ... .