In Samuel's lair, act 1, you do not get to kill Samuel, just the 4 guards. If you approach slowly, you may be able to get 1 or 2 to attack without drawing all 4 to you at once. You can sneak, attack with a bow or magic at a distance and withdraw to lure 1 or two away. You can also have a character take the stairs to exit the lair, then use potions, go check the BF merchants for better equipment, etc, before returning.
The act 1 summoning doll can equip a crossbow, so kept out of harm's way can provide some support (it would not last long in a melee fight).
In act 4, the third summoning test, when Samuel shows up he can not be defeated directly. Leave the summoning circle; when he tries to follow the summoning will be undone.
Later in act 4, you should be strong enough to defeat him. If you have the Hammer or Insect Swarm skills, those work fairly well.
If that doesn't help, what character types are you playing?
Welcome to the forum.
