That's a feature, not a bug. It is to prevent accidental movement of the camera when clicking on the skill bar during combat, etc. There have been suggestions to allow screen movement after a short delay, or something, which would still prevent most accidental movement.