That's no more than normal MAD for half-martial caster.
But yeah it's certainly unfair compared against what Eldritch blasters gets. That pact of the bladers require more invocation investment and yet still will be less effective. I'm not personally a fan of using Cha to hit, but I see little other recourse to even out the difference.