Honestly, just had to experience this. I didn't realise my game was set to public so had a random person come in and join. I couldn't figure out how to kick them, so I quickly saved and left. When I loaded the game again, I saw that the character the person made was still there.
Luckily I had an earlier save, but this is incredibly frustrating. I also felt like maybe there should be an indicator that your game is public other than settings, because I had no clue the setting was on until it happened.