There is two other ways to enter the Underdark besides the temple. And you can take the mountain path to skip underdark completly.
If you really want to use the temple entrance though, try stealth positioning you entire party in high ground and it should be a lot easier to kill the goblins.
