I agree that the skip dialog needs to be a different button on controller/console or at the very least change it to be HELD. The amount of times I have accidentally selected a dialog that I did not want because I skipped but got the prompt to select what option I wanted in dialog is highly frustrating.
Please consider fixing this as part of the newest patch Larian!