I also hate that I keep accidentally clicking on one of my party members whenever I'm just trying to move around. Drives me nuts.
And how do you feel about selecting a character just to cast something and having your entire party re-shuffle their positions for no reason? Double fun if there is a ladder involved.
