I think it's 1 hour real time OR level up. So if you level up midway through a 1 hour period, it resets the 1 hour.
In the script a "gamehour" is 5 minutes long and an NPC will refresh after 12 game hours. 60 minutes.
But also checks when treasure was last generated and checks if there have been more than 12 gamehours since then. So that count should be reset on the level up trade gen. Thus it's either 12 gamehours or on level up.
Also this is all on a per NPC basis, it doesn't do them in a batch.
At least I'm pretty sure that's how it works, there are others who are more familiar with it than I.
I can guarantee that is is neither of those things. Well, they may reset every 60 minutes real time and on level up, but they also restock more often, seemingly randomly. I experimented with this once I got to the first town area, in the warehouse. After the guy guarding the door restocked when I went to sell some things to another person since he was out of gold and I lost a nice pair of boots, I started to quicksave and quickload to test some things out.
Sneaking and looting the chest nearby seems to cause him to restock. I loaded a save from before and waited a few minutes and he did not restock. I loaded the same save, looted the chest, then went right to him, and he had restocked, so it wasn't simply an issue of just hitting the 60 minute mark.
He also seemed to restock when I traded with other people.
Another interesting thing to note is that when he restocks, it isn't random. His inventory always changes to the same thing. I loaded the save from before I looted the chest, checked his inventory, then loaded the save again, looted the chest, and checked his inventory. I loaded the save again, looted the chest, and checked his inventory, and he had the exact same things as when I looted the chest the last time, but different things as when I went straight to him.
I REALLY could use some clarification on how this works, because it's extremely frustrating to come back to a vendor a minute later and have an item you wanted to buy disappear, and have him have different goods every single minute.
I kept trying to repeat my steps and get those boots to come back into his inventory, but I can't seem to get the right set of goods in his inventory. I've managed to spawn about 4 different sets of inventory for him, all within 60 minutes of real time, so I know there's something else to this than vendors simply restocking every 60 minutes and when you level up.