So it looked like every cellar dungeon was connected, I could leap in to other dungeon!
It does seem like locations are shared on the same map. For example, when you head down into the lighthouse cellar and talk to the ghost about the undead base (which he says isn't located there), you can clearly see another indoor location off to the north of the lighthouse cellar map.
