Also, it's really easy to get to level 4 quite quickly.
That is intended tho ...
Take it some classes are lame only on levl 1 ... but some others keep at it until level 3.

To make level 5 you "should" get double as much xp as you get through levels 1-4 together.
To make level 6 you again should get "almost" double as much as for every previous levels ... and so on.

It will slow down noticeably.

Just those first few is so incredibly fast.
Remember, as you level up you face stronger enemies that give you more exp. So even though "to make level 5 you should get double as much xp as you get through levels 1-4 together," it doesn't actually take twice as long. A CR 3 Creature is 700 XP, a CR 4 Creature is 1,100 XP, and a CR 5 Creature is 1,800 XP, so you're almost getting ~1.6x the exp per fight as you level up.
The really low levels should be relatively quick, sure, but at least levels 3->4 should still take a comparable time as higher levels. Especially since many classes get so many abilities at level 3 - subclasses, level 2 spells, etc - so spending more time at level 3 wouldn't be bad.