I've found the base of four to be pretty good, it allows every character to contribute something to the fight and feel like they're participating. When I first started playing the first game, I wasn't really sure how the stats and AP worked and my first melee character felt useless in a lot of fights because he simply couldn't move very far, couldn't move and attack and/or couldn't use enough abilities to be useful in the engagements.
While the above problem was remedied once I began to understand the correlation of speed to AP and began thinking of my turns in terms of AP efficiency, the base of four and the cost of actions and abilities in this game has made building a character a little less dependent on min-maxing the build to make it effective, which I encourage.
On the other hand, AP is still a premium and the more you have, the more effective you are. I do think having a return of some way to increase max AP and AP gain per turn would help accentuate the "speed" style of play.
Just an idea but maybe a counterbalance of some kind. Like, a player could build a character to have and recover more AP per turn allowing for more mobility and actions per turn but this kind of build would require a sacrifice in some other area (defenses, damage output, durability, etc) while another player could build to be more efficient with their AP and accomplish the same tasks with less actions and movement while a third player may be trying to find a balance between the two? I'm not sure if that's possible, just an idea.