Khaga's key is for her footlocker not a door. I don't quite understand why you would be having her telling you to "hunt them down", I did not have that issue when I did a kill the grove run. If they are behind the stone door in the center of the grove ritual area, you can try going the back way.
misty step or jump to the ledge in the back door tunnel
