"I can get around this most of the time, by going to the fort, going up the side ladder, recruiting anyone I want in there, and then going back to the beach."

I've found that I can't do that with Ifan because Elodi replies and you level up! Is there a way around this please?