Oh, well I am terribly dumb. I thought we werent even able to get on the boat in EA?
Minor spoilers but still
In EA the boat crashes and all the act one is about leaving the Island were Fort Joy is. You finish the EA by leaving on a different boat. The pyramids are on that second boat.