Yep, I've had no more issues with DX11, so there must've been something with Vulkan in that area. I'm going to hazard a guess it has something to do with distant objects for one reason or another, since I had it only when looking around.
For completeness sake: I'm running on highest graphics with an RX6600XT, with the latest drivers installed. Windows 10.