Missed the bit about 4 people sorry.
Hmmm, perhaps the experience is getting spread too thinly across four party members then? Have you travelled to the Homestead place after talking to the Imp fella? Following that quest chain pretty much got me to 5th level with three party members. And have you spoken to the Legion Commander about the Lighthouse and Church, etc? Sorry to state obvious as I know you said you've done almost everything within the walls.
