Fergus was level 4 when you were level 3, so the level 4/5 difference should not be impossible.
I equipped my hero with a bow and stood at the south side of the corridor across from the cell. I opened the door with my DK warrior and attacked. A couple of the ghosts' hit points got quite low just as my DK's did, so rather than use a potion I ran away and let my hero kill them the next hit or two with an arrow. I think the other couple ghosts required a potion each for my DK.
Like Srikandi said, if you take out at least the left spider in the arena, it is possible to get the first battlefields key to level up.
