The pit is just south west of the graveyard, on the western outskirts of the village. The imp who asked you to save her brother should not have wandered too far from that point. However, the BF key in the pit is only level 2, and that dungeon would have already been opened by the higher level keys you have already found.
I think you need to click on the odd gems in your inventory for them to trigger a fire elemental. I did the same as you and just carried them around for awhile; IIRC I had to reload an old save to test them (don't recall if they became inactive eventually or if I had moved on to the next act before discovering you needed to do more than just pick them up).
