Fergus isn't all that hard if you follow a few basic rules:
1. Talk to the skull on the table before the fight (won't help in the fight but afterward you will be glad you did).
2. Make sure you have assigned all available attribute points. At this point in the game agility is where you want points more than anything else. You should be level 2 on normal difficulty or level 3 playing hardcore.
3. Also assign available skills but this is far less important.
4. Get a better weapon if you can - the knife on Fergus' table is better than a club - if you don't have better have your hero grab it as the fight starts.
5. There are healing potions around - they do not take any time to use and this is one of the fights that they are in the game for - don't save them, use them.
6. There is a picture on the wall. If you click on it a doorway opens - leads to a shrine that will replenish your hit points once and your mana once. Makes the fight easier.
Obviously you have already gone down the hole under the blanket. If all else fails if either of your characters can make it before someone dies go back up. Fergus won't follow and you can regroup and return.
This board has a search command - use it to find threads on how to kill Fergus (but you don't really need to this should be plenty - the average game I kill him without even needing a healing potion but LOL having done it 100 times or more does help).
Reality is for those who cannot cope with fantasy. Fortunately, I can cope quite well.