I haven't tried it in multiplayer myself so I can't give you a 100% answer. But there is a text box for subtitles, dice rolls, flavour text etc. which probably also covers chat unless there's also a separate chat window in multiplayer. Since Divinity: Original Sin 2 had a chat text box, and BG3 is heavily based on it, BG3 also most likely has one in multiplayer.
