Perhaps a "you can't quick travel with enemies around" solution.
But then again if that were added, how close is around?
They could also disable teleporting in certain structures?
Which could suck if the ladders are all destroyed haha.
I think the most balanced solution is that you can teleport to a rune only if you are, let's say, lesser than 5 meters from another rune.
