Where and on what are you doing your testing? Some enemies and objects have resistances to specific damage types in which will subtract off of your damage total. You can examine enemies, objects and NPCs by right clicking and pressing examine. So even if you roll that minimal 1 and get that + 3 added from your ability bonus, they might be reducing it by 3 damage from resistance, right back down to that 1. You can also check this in the combat log if your damage was reduced. The reason you've never seen your Eldritch Blast hit a 1 is because (so far as I remember) it's force damage and nothing has resistance to it. The tutorial area is full of enemies with resistances, so it's not a good testing ground.
