In act 2 once you get to the imp village, the first time you leave you will be summoned away. Don't leave by trying to enter the BF, then enter and leave the BF from the new location (you will miss the summon quest that way). Walking out of the village, being summoned and then entering and leaving the BF is ok.
The end of act triggers (after which the game becomes linear leading up to the end, and other quests can not be completed) are;
act 2 - giving 3 items to the alchemist
act 3 - accepting a certain witch's challenge
