Huh. That's interesting and sounds like some kind of bug. For me, every single time I've escorted him, I instantly have control as soon as the conversation ends, and I can start moving well before he does.