Going down with XP pts is due to game balance - the higher your level and the lower the enemie's lvl is the less skill pts you get. At lvl 34/35 I only got XP from Giant elite orcs.
That's why I killed every monster in Rivertown before entering the Castle - and did the Dark Forest afterwards (stronger enemies, higher lvls). And had end lvl 52 at the end of the game (maxed out wisdom, too).
Kiya