Found a way to fix it camera rotation stutter. It is enough to reduce the frequency of updating the mouse to a minimum (in my case, from 500 Hz to 125 Hz) and the camera became much smoother.
It looks like the game does not know how to synchronize the mouse control and too many commands from the mouse, it simply does not have time to process and misses, and as a result there are sharp jumps when turning the camera.
