Due to its size the door in the harbour building doesn't look very much like a door. I missed it my first time past, but after checking out the area on that side of the building I went back to talk to the fish merchant before taking the elevator platform up, and just happened to highlight the door on the way back.
