There is currently a bug with the secrets marker than can remove them after you save and reload or switch maps. Until this is fixed in the next update, you can place a manual map flag on top of the secret marker when you first read the treasure map.
You still need perception to find the exact spot, though, or trial and error digging up the general area (or a walkthrough). There are potions and spells you can use for temporary buffs, as well as possibly equipment bonuses.
