Yeah, there must be a bandits cavern in the Orobas Fjords. Some bandits got stuck there because of magic barrier (a trap). You need to deactivate that barrier and kill them.
You will find the book there, on a desk.