Great thread and good points. I especially like you going into depth about unsatisfactory solutions and why they are so.
Larian is aware of this issue. I'm not sure what they'll do about it though. There was talk about making Rank 5 more attractive, but that doesn't solve the whole problem, all the ranks have to have some incentive. Larian is moving away from AP cost reductions because as you pointed out, it is a balancing nightmare.
Your idea sounds pretty good, and it fits in with the AP penalty for weapons, but given the AP cost of spells in general, I think that a 1 AP penalty per rank above your specialization is sufficient. 1 AP is quite a lot, and even only a +4 AP penalty on Rank 5 spells will probably be quite a lot once you factor in the spell's base AP cost.
EDIT: Maybe, if that's not quite enough, Rank 5 skills could have an +2 penalty instead of +1.
As for the 7 tiers of skills versus 5 ranks, maybe rank 4 and 5 have two ranks worth of skills each, instead of giving rank 5 all three ranks.
