It makes sense to work as it is, aside form maybe the fact that it can apply to other characters than one with no penalty. It working on one character with NPC party members makes sense, and is in no way broken.

I personally LIKE the fact that it works as it does, in fact, allowing you to really have a 'main character' feel while still having minions.