Funny how I see ppl complain about things calling it unfair instead of thinking before acting in a fight. Have you thought about you know, killing the sleeping ones first? It's kinda sad someone blames unfairness towards the player just because you're inexperienced or didn't think before acting.
Remaining close to any edge is not the best move, players or enemies, you will be shoved at some point, and if you may not know, dash consumes action, shove bonus action, so it's quite normal, and I like that they can think on using it. It's your fault for not being careful.
As for enemies joining, there are patrols, if one gets close that will alert the others. I never once died or even lost a party member at the Goblin camp. Just explored it first, found some small holes, covered them, killed all sleeping ones, done. Just shoot them from high ground. Only a few may manage to get close now, but easy kill.
games like this and DOS2 will punish you if you think is just go straight to battle like most RPGs and that's it. Thinking, planning before acting is what will make this game easier and funnier to play. Every time I find new ways to approach battles, and that's what makes me replay it so much.
I understand that you may feel frustrated sometimes, but after you start seeing things like it, and spend some time planning, positioning your party members before the beginning of any battle, you'll see how amazing the combat can be.