I found a Oathbreaker in the first act in the camp. Maybe he won't move with you and will stay where he was first invited.