We haven't changed anything with engine in recent builds. Very low FPS = very low video memory. In your case it's very very possible. You have 512Mb and huge screen - using High textures will put your video down to its knees very very quickly. As to why it fluctuates - the reason lies in OS uptime, running apps (like Safari), and number of monitors attached.
I can very much firmly say that 512Mb on OS X 10.9 is not enough for High quality textures in 1600+ resolutions.
