No, since all those already have their own difficulty in finding the quest (perception/map or puzzle). No point adding another layer on top of that.
I'm fine with other chests found along the way, but, again, I barely found any of the kind.
I wouldn't object to some chestkeys being gone, or the chests of the graveyard (Not limited by perception or map) to be lockpickable instead of that not paying off key and chest 'puzzle').
