You can fix this by using subsequent bonus actions to go to stealth away from all the bullshit.

Eventually this will be nerfed, but maybe eventually concentration will be just a tad more reliable too.