2 ways i have found to get inside without being attacked. Rescue Sazza from the Grove, she is in the cage or just talk to the Goblin near the entrance and try to talk your way in.