To be fair he's not bad as an eldritch blast spammer, but he needs a light source or the darkvision invocation when he's underground. I think most people's bad experiences with his accuracy comes from him having disadvantage when spelunking.
When I'm going to use him in the full game I'll probably give him +2 Cha at lvl 4 and Moderately Armored (+1 dex) at 8.