I’m not quite sure I’m understanding what you are describing, but …
IIRC there are enough gems to collect. One for each of the three Sharran trials, one from the shielded area where you enter (is that a gem that you collect once the shield is down? I don’t recall for sure), and one from somewhere around where you find Raphael’s enemy.
Have you done all of that stuff?