Those permanent rocks are just decoration...
As for holy water, you can get one as quest reward from the imp hermit in A2 and for the rest you'll just have to hope you find them in the bf;
it's purely dependent on luck (as in YOUR luck, not the luck stat of your chars...)
