They said switching speakers mid-dialogue would lead to too many permutations to be able to do.

That's fine and understandable. No one is expecting a 5+ people conversation that can go anywhere. But not letting party members to use their skills is not ok. Gale could easily interject with a History skill check, which would then give the speaker the knowledge to continue on that branch. There are many situations where everyone in the party should be allowed to roll a skill check, even if only one is doing the talking.