You didn't break the quest any. Had you waited until now to talk to him, he simply would have claimed to have lost the key and kick you out of the house, leaving you in the same situation you are in now. To use the key on the door (or just to see doors on the northern sides of houses) hold the alt key down (waving the mouse cursor around should also find it, though slower). IIRC you can Lockpick the door before you confront him a final time (after which it gets sealed), but that will leave the key in your inventory.