Yes that key works for both. You need to go back and find another save game from before you did that, use Raze's hex editing system to give yourself the key, add in the key to your inventory from someone else's account (again Raze for that), or just keep playing. The Garden is not necessary to complete the game, you're just missing out on some gold and a quest or two.
