This has been a problem since early access. Mouse camera panning is turned on, but it doesn't work unless you move the camera with the keyboard. If you switch characters, you have to nudge the camera again with the keyboard before you're able to pan using the mouse only.
