Bought the game yesterday, and I'm having the same problem. Also have other friends on Xbox Series X/S with the same issues
Sent a ticket to Larian, hopefully they'll answer or realize this is happening to a lot of people and fix it