High up on the southweshern cliffs in the Goblin city (found in the southeast of Orobas Fjords) is a cave entrance. Svadilfari is in there. Whether you destroy the goblin forces on the surrounding cliffs first is up to you, however you'll probably fare better if you destroy them.
