There appears to be an entrance at the top of the castle/mountain area you may be able to access to. If you want to avoid the fight I would rescue Sazza, it leads to some interesting options later on.