Yes, I agree with this. Also there has to be timing involved in how long the rider stays there by the bridge (maybe 8 sleeping sessions as an example - after that he leaves and only the raiding party is encountered), because if you avoid the scene and comeback after you are lvl 10 and can maybe manage to kill him that would just be bad. I do think he should use breath weapon and fly away if you poke him, likely killing lvl 4 characters if not the remaining party of githyanki can do the job.
