I honestly feel like a party of 5 would be the sweet spot. As it is now is absolutely too limiting, especially if you're not super good at the combat and you need the extra security of having a tank and a healer. I for one am only okay at the combat and I just would not be able to manage without a tank and healer at all, even with them I need to constantly heal with potions and food. Also here's something important to note:
The goblin camp has stuff specifically pertaining to Wyll's story and if you go there without him you miss out on it. Which means if you want to experience his story then you have to take him with you and thus you only have two slots free, limiting the party interactions and reactions that can be had within the camp.