The issue in this case being Larian's very, very poor decision to only have one skill vendor in the act who sells Pyro, Geo, Hydro, Aero and Poly skills.
The issue being that since the final fight of the game uses different rules, I wanted to change my skill setup to take advantage of those and use skills I'd otherwise never bother with. Except now I can't unless I'm willing to go back a few hours and redo a bunch of stuff - which I am not.
On the other hand, you know what to do on your next playthrough.
Having done the final fight recently, I disagree with "Different rules", not even sure what you mean by this. You won't be able to do the strategy you wanted, but you can make do without any of those skills, at least on classic difficulty. I finished with a sub-optimal party with a worse set-up.
Sensechut is right about Geo skills. You may find some other skill books as you play through as well to complete, the only one I had issues with was polymorph, I fnished my elemental books on the nameless isle (Which is why I really don't think it's as drastic as an issue as you claim).
I get it's frustrating from a completionist's standpoint, but you're really not meant to play the game in this way, and more so embrace the RP aspects. Could it be better? Certainly. Move the vendor closer to the area you would travel before going to the demon's house or have it mentioned in dialogue, but if you haven't met him, it won't change anything.
I personally like the consequence, I think it's fantastic, and wouldn't have it changed, just the approach.