Take Will in your party from level 3, have him summon the imp (turn invisible) and send it off to explore the goblin camp... Once it finds the portal just teleport the party in past the guards at the bridge
Luke Skywalker: I don't, I don't believe it. Yoda: That is why you failed.