Are you sure the process of switching to the other character isn't simply auto-completing the mini-game for you? Because there is an option to skip the mini-game and just resolve the conflict with a behind-the-scenes dice roll.