While playing with a controller, can't you just swap back to them and cancel end turn? It was a problem in EA where you can accidently skip characters turns by hitting space too many times but that seems to be fixed with the ability to cancel end turns.
