I have exactly the same problem, everything stops for few seconds, I can still move mouse, sounds like music and dialogues still plays. It can happen at any time, sorting inventory, middle of battle or exploration. I mostly noticed it around fort Joy, didn't venture forth yet, what's more I didn't notice it during tutorial ship, and I took my time exploring it.
I also later tried making new game and rush to Fort Joy on easiest difficulty and it happened there as well.
I have pretty bad specs, Radeon HD 7750 GDDR5, E8400 Dual Core, 8gb ram. Using HDD too. But the game is generally well optimized and other than long stutter I have 40-50 fps in overworld, inside caves it's stable 60fps. (it's limited) Battles work great too.
Also question to the OP, can I ask you about your specs (especially GPU, I've seen someone in the past who had similar bug suggest it might have be an issue with AMD GPU's) and if you overclock anything? I have gpu and cpu overclocked, I'll try to remove it and report if it changed anything.