I had to remap my keys rather than using middle mouse button - E and Q are now my camera panning keys. It's helped quite a bit, but it's still not nearly as fluid as it should be. Just port over the system from Divinity 2 maybe?