I might be wrong but I don't think this is timed but triggered by the Absolute finding us when we enter the Goblin camp. If you never have the moment in which you are overpowered by the Absolute's presence and saved by the prism, you can explore the whole Underdark without meeting your Guardian. Up until this point we have also been lost, Edowin and his siblings were charged with looking for us.
I am not sure what happens if you completely avoid the camp.
