The people of BG3 cannot recognise him as they don't know him. He ran off into the wild when he was a teen. Top-level aristocrat tweens are rather unlikely to be known by sight to the wider Public. "You Ravengard's son? Yeah, and Elminster's my daddy. Fuck off, druggie!"
