My concern is that making Daggers about being very effective on those without armor and very INeffective on those who do have armor is that what happens in fights when the only targets are those who have armor? Then the Rogue isn't going to be able to contribute very much
Simple! Give them more kinds of weapons or give them alternatives.
Also once again ALL targets have armor. ALL OF THEM!