I find the combat difficulty just about right personally. After my first playthrough, I only encountered about 4 particularly difficult fights. Larian games are kind of like From Software games. There's a trick to them, and once you learn it and everything clicks, it becomes quite manageable, without usually ever tipping into easy.
Other comments have already offered some good suggestions, but I'll also mention that food is your best friend. Most food will heal you a small amount, and you can eat it as a bonus action; some rare food can heal better than potions. I was absolutely slamming apples and sausages into my character's face every turn that I had a leftover bonus action, which as a fighter was pretty frequently.
Also, something to keep in mind, particularly in that temple fight, is that light affects your chance to hit. Attacking enemies in shadows is disadvantaged, and if you're playing a character with darkvision you might not even notice that they're standing in a shadow until you switch to a human, or pay attention to the hit-change tooltip that will give you a warning. Make sure your cleric or wizard has dancing lights, and pre-cast it before the fight if you can, or just cast light on your fighter. Or, if you do not want to commit concentration to a maintained spell, light some existing light sources in the room, or drop lit candles in the combat area, etc.
You should be able to get to level 2 before ever getting to the temple though just by gathering your party and exploring a bit. I got to 2 on my most recent character just from killing everything in the tutorial and gathering 2 out of the 4 companions.
