Cazador is tough. Come back when you're level 12.
If you just disconnect Astarion from the party and initiate dialogue with the other 3, you can begin the fight without Cazador getting the ritual buffs. Then, just have Astarion come down the stairs and join the fight. This still gives him the final confrontation with Cazador and makes the fight infinitely easier. I always made sure to level 6 counterspell Chatterteeth's eyebite as well.
