I found that i you go where part of the upper tower have fallen onto the ground below there is a network of ladders leading up more like a back entrance as I could not use the front entrance either. Once you spoken to the wizard follow the road to the water cross the stream by either foot and carry on heading north west eventually you wil come to a small drop go down the drop and continue to go forwards and you should see the ladders and platforms you can clime. Hope this helps
