(on a weekend, 5 in the morning in Belgium!)
I'm in Canada, not Belgium (though it is after 5am here now; I got up and had to check the forum/email on my way back to bed...
). I work evenings and weekends, since that is when more people play games (and working late I can overlap with the morning in Belgium).
Not all items have a handle, only global items. Picking a local item up and dropping it will create a handle for it.
Deleting a handle does not allow it to be reused.
I don't know about the removed items. I can ask about that, if it matters in light of the previous answer.
The handle generator uses two parts, one of which is a global counter (which only increments each time it is called), and doesn't keep track of or look for handles that are no longer used.
I don't know about ItemHandleMoveTo.
AFAIK local items don't use a handle.
I was told this could not be fixed for D:OS, but that would be changed for D:OS 2. I don't know if that was purely for technical reasons, or the chance of major changes breaking things for others was too great a risk given the relative rarity of issue with the higher limit.
Canada, interesting. Did Larian specifically pick someone living overseas for the job so that 'normal' working hours would overlap with european playing prime times ?
Thank you anyway for answering questions on a weekend !
So what you say means that the engine never does some kind of 'handle space defragmentation' which inevitably leads to death at a certain point, although there might still be a lot of handles potentially usable, but the engine never bothers to look for the 'holes'.
A handle needs to be created at the latest when the item is put into some container, independent of the item being global or local (globals might have a handle created on game start to be fully prepared when Osiris uses them). Otherwise we would not be able to get the handle in Osiris 'events' like ItemTemplateAddedToCharacter( (STRING)_ItemTemplate, (INTEGER)_ItemHandle, (CHARACTER)_Character ).
All these 'events' are queued as soon as an item is put to any container.
Which makes me think, that it's actually a slot that can hold an entity (a single item or a stack of items) which aquires a handle, attaches it to the entity and sends info about all this to Osiris. It also sends who or what its parent is, so Osiris can categorize it as either an ItemTemplateAddedToCharacter() or ItemTemplateAddedToContainer() 'event'.
I was also wondering if the change to treasure creation from classic to EE is at least in parts responsible for the overflow crashes (which I had never heard of before in classic), because with the early creation on zoning instead of 'on-demand-creation', a lot of handles might become unnecessarily wasted, which could lead to the limit being reached much earlier than before in classic.
The fact that people (me included) had crashes on zoning also suggests that the new treasure creation is the cause for the overflow.
Of course without ever 'defragmenting the handle pool', this problem could have occured in classic as well, but much much later.
(If it works the way I decribed above - treasure generation puts item into slot, slot attaches handle.)
I think we all know, that it could of course be fixed in existing DOS, but in a 'time-is-money' world it doesn't make sense economically, because noone will pay Larian for it. (The vast majority never ran into this problem after all, myself included after my 'private changes' ;-)
I've heard that D:OS2 is even biiger than D:OS was, so it's a pure necessity to fix this weak point.
Using int instead of small could be a possible fix which would push the limit much much further, although not solve the principal problem.
Maybe the last fix that doubled the limit was not even a change from 'small' to 'unsigned small' but was simply changing the generator's counter to start counting at -2^15 instead of 0, easy fix which would effectively double the amount of available handles. (Much easier than changing the data type. The value itself is not of interest anyway.)
If handles are handed out by a 'handle generator' it would in theory only be a matter of changing the generator to search for 'left over holes'. Any bookkeeping of 64k handles needs only an 8k bit field, which can easily be stored in a savegame and has a chance to even be permanently kept in modern processor's caches. Of course this would be dependent on handle users informing the generator about releasing previously acquired handles, which might not be the case currently and the reason for 'technical impossibility'. The save process not being able to store a generator's bookkeeping bit field might be another, maybe there are problems storing binary data in the save XMLs ;-)
Anyway, bookkeeping and reuse of handles is the only way to go, every operating system would be unable to run for a longer time if they never reused values of released file handles. Blowing up the data type for handles only pushes a limit further away but the limit will be reached at some point.
What you said about the generator - counting up, no bookkeeping ... crash on overflow ;-) - means, that we can never make any changes like deleting handles to prevent overflow, so information would be helpful, if we could find any way to prevent unnecessary handle creation in the first place.
'UsePartyLevelForTreasure' for containers comes to my mind, maybe setting that could prevent standard treasure creation and cause a fallback to old 'on-demand' creation for placed containers at least. More info could also tell modders, how dangerous blowing up treasure tables could potentially become.
Apologies for bad grammar and english, it's not so easy for us non-native speakers.
Maybe you will eventually find the time to 'dig deeper' in Belgium, I'm sure modders would do as much as they could to help soften problems if possible, but they need information. And we'd love you even more than now already ;-)
