I had shields equipped for both. As soon as I got rid of the shields,weapon damage almost doubled and, I was able to take them out. Many of you said I was supposed to sneak by them. That did not work they saw me and attacked anyway.
This is because you have invested skill points in onehand weapon, you need to invest in the skill one hand with shield if you want to fight with a shield.