Valhalla, if that helps, I have almost the same configuration (4070, i7 12th). I encountered the same problems, and updating nvidia drivers (even though I already did last week) made it up.
No mods, but on level up it now takes ~15 seconds of loading before the level up menu appears, and will sometimes freeze on a black screen with music playing either before or after the level up.
Loading times - without seeing a loading screen with art, if that makes sense - have gone from near enough instant to 10-15 seconds as well.
That's on both Vulcan and DX.
Haven't tried a full wipe and reinstall, although I'll probably try that later once I've had my fill of Starfield for the day^^
