The entrance to the mine should be marked on your map (
Broken Valley map); it is a little south east of the quarry tower (where you rescued the guild master and others). The other tower that you can not enter is the top of Lovis' tower, and there is nothing there but Dr Needleman (he is missing his hat, so can be mindread to get the password for his storage area, where you may have been teleported high above by Bellagar earlier in the game, though there is nothing of any great value there).
Welcome to the forum.
