I have not played co-op with the alpha, but to host a game, just click on New Game from the main menu, the same as for single player, and go through the character creation for both characters, then in the game hit Shift-Tab, right click someone on your friends list and select Invite to Game. Or was that what you were doing?
If you click Join from the main menu you should be able to join the game of anyone on your friends list.
Do you have the Steam overlay disabled? If so, enable it in the Steam options, or allow it just for D:OS.
