When you go into the teahouse in the swamp, Astarion realizes that he can enter homes without an invitation. But my party met the old woman at the swamp entrance and she invited us to come to her house, so that would count as an invitation for Astarion wouldn't it?