Well I haven't tried it yet but I think I figured it out thanks to your help with the hint of talking to ghosts.
*spoilers*
There's silver ingots near the shore near the empty vault south east on the island. You need to turn those into levers somehow with the forge on the island.
