I do understand your idea and I'm totally against it, just as others mentioned. It would totally screw with mages, because they hardly need those battle skills, they need to be able to level more skill trees.
Also it limits options instead of giving more, defying the concept of the game: Play how you wish to. Just like splitting of social skills did.
Physical damage is op not necessary the rogues themselves. If you put as much effort into the warrior as you put into the rogues damage, he probably would not have been not much worse. Comparing the damage of somebody who had only 4 AP with somebody who had had how many, I guess 7, at his disposal is a pretty flawed comparison. Physical anyway depends on weapons, for a real comparison both would also need comparable weapons.
