That is working as intended. You get a message about that the first time you pick up the second pyramid.
You can avoid the pyramid being put back in your inventory (for now) by placing it in a container, or leaving it with a henchman/companion not currently in your party. However, that can cause problems if you use the other pyramid on another map (you can teleport outside the level bounds, or otherwise end up somewhere you don't want to be).