Maybe a second delay, or right click (etc) with the cursor at the edge of the screen could unlock the screen, so people will not be accidentally moving the screen focus, but it will still be possible to do so with the mouse regardless of the UI elements.
I'd say just add a slight (like, half-second) delay to the panning if there's a UI element there. That way the edge of the screen is still easy to hit with quick mouse movements, and accidental mouse movements won't pan the camera, but if you want to pan it it's easy to do.
"But I don't want to go among mad people." "Oh, you can't help that. We're all mad here. I'm mad. You're mad." "How do you know I'm mad?" "You must be, or you wouldn't have come here."
