I had occasional slowdowns with a lot of trees or NPCs on screen, but not the reverse.
Are you shutting down all non-essential programs (especially anti-virus) before starting the game? Maybe a program running in the background is slowing the game down most of the time, and sometimes stops what it is doing, allowing the game to speed up.
