Did you try zooming in or hitting Alt (X on Linux)?
You don't actually need to keep the inert stones, though they are a handy way to keep track of the number you have found.