You lose them just before you arrive and don't get them back until you cannot access the House of Madness again.
Or am I wrong? I haven't reached the point where you retrieve your pyramids. Can you re-enter Iona's Dungeon after you retrieve your pyramids? What if you left the House of Madness alone until then?
