I believe you have to be wearing the silver armour, just having it in your inventory probably isn't enought.