It's a bug.
The whole room is a little buggy.
For example, as you can enter it by going into the hole from the surface, you can take fall damage if you jump into the hole rather than path to climb down.
Also, after going down the hole, there's a chance that one of your party members (Usually Lae'zel for me) is stuck in the air and cannot move, they have to manually jump down from floating in the air before they can then traverse the room normally. (Which is slightly to the side towards one of the cabinets. I'd posit this is most likely the cause for your mysterious fall damage if your character is considered falling from touching this random invisible floor)
I've also had a couple of times where going down the hole randomly has my character downed as if they were killed by fall damage, even though going down the hole is supposed to be safe. (This is seemingly tied to being in combat if for example you talk the people outside away, then talk your way in the front door, then go back outside and drop the stone and go down the hole - The people outside the room will run in to investigate the sound and start combat with you)
As a side note: Items falling on a character do Crushing damage not Falling Damage. You can see this when you throw larger items (Such as the Returning Pike) and enemies take a small amount of Crushing damage in addition to the thrown damage, or when you encounter the bug with returning weapons where they can crush your character because the game's physics are written with spaghetti code.
