I was able to get to grymforge without anyone in my party or camp by skipping the goblin camp but once you get to the goblin camp, your game starts to get finicky.
The cutscene when shadowheart's stolen artifact saves you from the voice calling to you trips the game up because shadowheart (or anyone) is not in your party/camp.
Refusing her to join still will be ok and the game seems to play fine but once you get to any vendor to sell something or switch things around in your inventory the game just closes on you.
this is a very interesting issue, and a good data point. Please, if you can spare the time, submit a report to Larian's official bug reporting form as well (you can find it here:
https://larian.com/support/baldur-s-gate-3?ver=4.1.1.1233832#modal ). Include a description like this (just copy paste it if you want, but the more detail the better if you can), and also include a save file where the issues can be replicated or easily reproduced. This would be a big help.
