This could be solved by not letting the familiar hide as a bonus action. While the process might sound tedious, it's pretty easy once you get the hang of it and I was even able to solo the four level four gnolls outside the blighted village with JUST the familiar.
Even if you mess up and the familiar dies, you can just try again for free. The only case you wouldn't be able to do this is in an enclosed environment or there being so many enemies there is no place you can feasible go to hide even with flight and movement.
There is bigger problem here and familiars are only a side effect.
Enemies just don't follow you reliably when you hide. You can do similar shenaningans with companions, the problem is that enemies will not group out and "check out" the place they got attacked from
