You can. Karlach even has an inspiration for getting there without a lantern, but it is a bit challenging.
You will need necrotic resistance and lots of healing as you run like hell for the towers. And you need to go completely south so you get the shortest route.
