I think the biggest problem right now with the passive abilities is that they can be taken off their separate hot bar and then there is no way to find them again. That can screw you up, especially if you don't notice that it has happened straight away.
Yes, that's particularly annoying; it seems to happen most often when the game is slow to complete whatever animation it deems necessary in response to a hotbar click and you move the mouse before it has finished.
Dragging new items onto a hotbar is OK, as it is a deliberate UI action, but I think, perhaps, removing items from the hotbar should be right-click-and-confirm behaviour to avoid doing it accidentally.
So far, everything I have lost has been available under the "Spells" UI screen ( i.e. pressing "k" to bring it up ), even if it's not actually a spell.
