You can talk your way in without Ifan, tell them you want to trade, it does have a persuasion check but I didnt have any trouble with it, your other option is to kill the Lone Wolf in the Driftwood tavern and take his amulet, that too gets you into the door without a fight.
