The level BF key you find determines the number of dungeons which will be open. The only difference between entering the BF after finding the first key and after the second is that in the first case only the first dungeon will be open, while in the second the first two dungeons will be.

The only potential issue with leaving the lower level dungeons too long is you may level up enough in the main game that you get little or no experience for the lower level opponents. See;
Act 2 - not getting exp in BF

The BF dungeons are optional; I only used the BF for the traders.