Ah, looks like the problem was because my Public and Mods folders were symlinks. My Mods/Public folders were symlinks to folders outside the game directory I use to organize my mods.
When I deleted the symlinks and moved the folders in, it saved this time.

Looks like that didn't fix the "crash when exiting" issue though.