Happy to see positive feedback on this.
Your idea sounds pretty good, and it fits in with the AP penalty for weapons, but given the AP cost of spells in general, I think that a 1 AP penalty per rank above your specialization is sufficient. 1 AP is quite a lot, and even only a +4 AP penalty on Rank 5 spells will probably be quite a lot once you factor in the spell's base AP cost.
EDIT: Maybe, if that's not quite enough, Rank 5 skills could have an +2 penalty instead of +1.
The reason I thought 2 AP would be a good number is because of what I expect will happen later in the game. In all my playthroughs, I have easily attained 14+ AP per turn on each of my characters. A +4 AP penalty would not have been very significant.
With that said, I am not attached to any numbers in particular. I think Larian would have a much better perspective to work out any of those types of details.
As for the 7 tiers of skills versus 5 ranks, maybe rank 4 and 5 have two ranks worth of skills each, instead of giving rank 5 all three ranks.
Not sure how I didn't think of that. Seems obvious in retrospect once you point that option out.
I would still have primary attributes serve as requirements and affect damage and precision or chance of success - as they are doing now. I find that mechanic plays nicely. Feels good in gameplay.
I should have been more clear in my original post. Everything else about the current system would stay the same, including the primary attribute effects you mention.
What if specialization increased the strength of spells instead?
The jack of all trades could still do whatever, but with less potency. (same as fewer spells per turn)
The specialist would lack certain abilities, but have more power due to increased focus. (same as more spells per turn)
Same result but different approach.
edit: /rereading/ I guess you mentioned this approach already above, but I think people are just as inclined to specialize in order to feel they aren't gimped by missing a bonus as they would be if they feel they are missing extra action points.
A lower AP cost vs. higher damage are fundamentally different in several ways. I did a poor job elaborating on this point in the post.
1. Bonus damage is always beneficial whereas bonus AP is not.
For example, having 1 or 2 extra AP is useless if you can't spend it on anything and you are already regaining all your AP each turn. On the other hand, that 1 or 2 AP might be critical to getting another high cost spell off in that turn. This is a critical element of the non-binary specialization argument I made, and it would be lost if the benefit were damage rather than AP.
2. From a psychological perspective, not having the benefit is completely different.
When you cast a spell that doesn't do much damage, it feels very unimpactful (even if it cost little AP). As a result, investing that 1 point into the type of magic doesn't feel like it was worth it. However, a spell that packs a punch still feels very impactful, even if it cost all of your AP to use.
3. AP has a much more universal balance impact.
Most spells in the game have additional effects on top of them or even deal no damage in the first place. Often, these effects are the primary reason to use the spell, not the damage.
That doesn't reward specialization. Instead, this punishes non-specialized characters
Yes, you are correct. I feel like the general consensus right now is that non-specialized magic characters are way too strong, not that specialized characters are too weak. Creating an incentive to specialize does not necessarily mean we have to reward it, just make the other option less appealing.
