Been a while since I've gone there, I'll try to offer more detail. After crossing the first bridge on the way to Lovis tower, keep going right down the hill. You'll go past skeletons and eventually find black ring assassins that will attack you. You might have ran into them and avoided them.
If you look at the map, it is to the left and a little north of Lovis tower. I hope this description helps you better.
