I've recently played through Trials of Tav on Honour Mode and I would say that the main things that is needed to increase the challenge are (in order of importance):
1. Less predictability:
Knowing exactly what enemy you are going to face and what they will do is the singlemost important advantage I have as a player. As soon as something unexpected gets thrown into the mix that I hadn't anticipated, things become so much harder. Beating the bosses in Baldurs Gate 3 becomes easy, because you know exactly what they will do and how to set up. If you did what Trials of Tav does and spawn in random reinforcements that will force the player to react and re-prioritize their focus, that would instantly turn the difficulty up. Make enemies more variable as well with spellcasters being able to throw surprises at you.
2. Limit the alpha strike:
The BBEG should not go down on the first turn before they even act. Give them a ridiculous initiative boost, or stick a sanctuary or invulnearbility on them when the battle starts so that they always get a chance to act and do their thing. Again it's all about forcing the player to react and adapt instead of having complete control of the battle from start to finish. If first thing that happens is that a party member goes down, or is paralysed or something where you immediately have to scramble to repair the damage, then the pressure is on the player. Again if you add the idea of reinforcements from the first suggestion, the alpha strike may be able to clear all the minions on the first turn. But if you burn all your resources on that first turn strike, then you might not have enough to deal with the ones who spawn in later, and if you don't know what those enemies are going to be, you might want to save some juice for later. Make the alert feat a bit less OP and instead make the player also have to prioritize gear and choosing the elexir of vigilance if they want to act first.
3. Give the enemies access to the same meta as the player:
Things like speed potions, strength elexirs, arcane ward and weapon coatings are all very powerful tools at the hands of the player. But if the enemy was carrying those around and made use of them as well, there would be a level playing field. Imagine a Dror Ragzlin or Minthara on a speed potion, that also is guaranteed to get at least one turn to act by implementing something from the second suggestion. Or a goblin brawler that chugs an elexir of strength is all of a sudden a much larger threat. Larger enemies like ogres should be able to do what the throw barbarian does etc.
