Yes, this can be incredibly annoying and I don't think there's an in-game way to prevent it.

At the very least, locking your skillbar should prevent new items from being added to it. I hope Larian adds this small QoL adjustment in a patch.