The reason this happens is because of how they implemented click-and-drag (yes, this is the chain system rearing its ugly head again). If your mouse moves even a single pixel between mouse down and mouse up, it ignores the click. This is very easy to do by mistake as we all move our mouses very fast.
The fix for this is for Larian to properly check that your mouse moved more than a few pixels, like a deadzone, before it thinks that you're trying to drag.
In the meantime, once you understand what causes this, you can click a little more carefully =\