Some of them have a big anti-dragon field around them. This field distorts light and does great damage to you in your dragon form (as you have noticed already). You can shoot the towers through the field but sometimes you can't target properly.