Don't equip your shovel either. The game will auto equip it for you, then remove it once you're done digging. This could help with the mystery deshovelling (hehe).