Yep, that's what happened to me, too. They disappeared one by one. This can even be seen in the TB queue.
A second problem is that as a Berserker including intimidation, you can force the Ogres into your service once for free. However, this possibility is only opened if you a) start the initial conversation with the Berserker and b) talk about it without interruption by refusal.
If you have previously approached the Ogres with another custom character / class, for example, and the main conversation was conducted in such a way that the recruitment was rejected, you have the option of approaching the Ogres again with other characters. However, if one does so subsequently with the Berserker, this specific conversation option is missing.
In other dialogues (except always at the very first sentence dialogue start, e.g. as a Bard at the mirror in the Necroapothecary's basement) such a thing is possible, but not here. Please change this in favor of group cohesion when playing with custom parties.
Initial conversation with Berserker dialog option during the course of the conversation.
Follow-up conversation without Berserker dialog option after previously refusing to recruit the Ogres.