How are you calculating "3.5 on average"?
Since the minimum of a die roll isn't 0, the average is greater than half of the die size. For the simplest example, take a d2. The options are 1 and 2, for an average roll of 1.5
d6: (1 + 2 + 3 + 4 + 5 + 6)/6 = 21/6 = 3.5
d8: 4.5
d100: 50.5