I agree that there needs to be an option to toggle it.
I also think it should be more situational. The dirt appears and builds up much too fast just based on time. It's a little jarring. It makes sense to get bloody in a battle or dirty trekking through a swamp, but not just strolling around the grove.
Having places you can go to clean up would also help with immersion.
