On my initial playthrough, I completely missed the 'explore the ruins' quest. To my surprise, Withers showed up at my camp offering respec services and hirelings for a mere sum of gold. At first, it felt like an added mod, but I later discovered he's an integral part of the storyline.
I guess you could say he joined your camp Wither without your input.
