There are actually several ways to make it to the underworld. One is the beforehand mentioned. There is a secret hideout in one of the north towns (besides the one on fire). You gotta get to a hideout of a "merchant" crew; there is a lever behind an illusory wall that allows you to get to the underworld. The other way is by going through a teleport point behind an illusory wall at the Hog's hideout. They are both missable.
The "easiest" way to get there is the one by the goblin ruins for sure. I found the other two after I already explored it.
