After digging into the settings I've noticed that my core affinity was not set to all cores (I am using the Process Lasso to divide system processes,games etc between 2 chiplets on Ryzen) but rather to the last 12 threads, same issue with first 12 - It did cause crashes. When sI've switched CPU affinity to defaul all (0-n) threads it suddenly worked both for Dx11 and Vulcan, no need for admin rights either.
I use Process Lasso also, and I suspected it. However shutting it down did not help. Assigning bg3.exe and bg3_dx11.exe in Process Lasso to 0-23 also didn't help.
WAIT! I had turned off Process Lasso's "core engine". When it turned it on, along with the above .exe changes, it worked. Holy crap.
So how come it didn't work even after I shutdown Process Lasso? What is Process Lasso doing after I've shut it down?