Is it immune to all physical damage or non magical physical damage?
It says "non-magical attacks". I'm not sure if it explicitly means "non-magical damage" one-to-one, but I interpret it that way, and therefore magical weapons, perhaps even enchanted ones like Shillelagh, should bypass immunity, as others have said.
Once upon a time, D&D made a big distinction between swords and magic swords. As an example, you could attack a Gargoyle with a sword and to zero damage but if you had a magic sword, you'd do full damage. I haven't studied recent rules before playing BG3 but it's possible this explains the mechanic.
That's true at least for an AD&D example from the predecessor where it was the other way around:
Magic Golem.
