There are story reasons why the security is not particularly tight. Just as you are about to enter the palace, the door unlocks automatically for you. Cazador is expecting you to bring Astarion so he can use him for the ritual. You can get extra insight into Cazador's way of thinking if you chat with him alone first and then talk to Astarion at camp. Astarion says: 'He won't throw the door open for us, he's no fool , but there may be fewer obstacles in our path if he's expecting us.' It looks like he put a modicum of enemies in your path, thinking it would be enough to best you or at least weaken you before you face him. He's very sure of himself and thinks you stand no chance against him.
About the palace - I agree, it's too small and is positioned in a ridiculous location. It was clearly meant to be in the Upper City, but since we didn't get it, they awkwardly squeezed it in. I think there was a line from Astarion saying the palace is imposing on the outside but sadly we don't see that in the game.
