Does this mean a game started as Singleplayer / Offline can be turned into a MP game in-game?
In Divinity : OS? Yes, because there is no singleplayer/multiplayer mode. There is only drop-in/drop-out, and whether you as the player are actively allowing it or not at any given moment.
The only thing that singleplayer/multiplayer prompt seems to do is toggle your Options > Multiplayer setting between "offline" and "invite only/friends/public". If the setting is set to "offline" and you choose New Game > Multiplayer, it switches it to "friends". If it's on any other setting and you choose New Game > Singleplayer, it switches it to "offline". However, you can switch it back to whatever you want by simply visiting the options menu or using the gem at the top of your minimap.