I'd agree with the OP, selecting another character when the inventory panel (or character/skills/talent panel) is open should change the panel. In combat, you should only be able to access the panel of the active character - attempting to select another character there should give some sort of negative feedback (like highlighting the selected character in red or popping up a message).
Not sure I'd rate this as a "must" though - stability and speed improvements would be higher on my list (recent patches have improved things, but game loads take longer than they should).
