I agree. The way I've balanced Berserker at my table is by having them lose a turn after rage is lost, the same thing that happens upon the loss of Haste. Perhaps they could do the same, not that it would matter with how short fights are right now. I am glad to see the subclass finally being used, but it is truly broken without that drawback.
