I am still stuck in the prison area trying to figure out if it is possible for me to reach level 4 before leaving the prison and facing level 4 enemies, and which fights are possible for me to reasonably do.
Yes it is but it is tiresome. Go and kill some silent monks at XP200 a pop. It is slow but you always win as they have a disadvantage of being blind.
As you have lost your collar you can go and pick up some source points from the frog cave and by the turtles.
You have a massive advantage when you get to level 4 - all the decent skillbooks are available from vendors.
You can even get to level 5 before leaving the fort but you need to be a bit unpleasant to do it (kill everyone basically)