Go into options / gameplay /" bottom bar options" and uncheck everything there so it doesn't auto add anything to the hotbar. Then go to the hotbar and simply grab the items in there, drag them out and let go to get them out. Now that everything is cleared out, open up up your character tab which shows you all your abilities, spells and inventory and drag whatever you want down to your hotbar in whatever order you want.
I do definitely agree that the hotbar needs a lot of work. Pop ups with tabs like Pathfinder Kingmaker uses are a really nice design and almost negate the need for a quick bar.
