Ladders should be useful choke points.
1.) You should be able to block off ladders by positioning a character at the top/bottom of ladders
2.) It should take movement to climb ladders. I'd prefer half speed, but even full speed would be a vast improvement over now.

This would allow interesting gameplay around ladders.
- Shove someone off a cliff? They can't just spend their 30 ft move to walk 20 feet, climb a 20-foot ladder (for free), and then walk 10 feet to attack you.
- Put ranged characters on top of a cliff and your tank at the bottom of a ladder. Now your tank can actually do their job - preventing enemies from reaching your squishies.
- Character ends their turn halfway up a ladder? They can be shot at, maybe even falling with a failed Str or Con ST.

Climbing walls in general would be cool, although this would need to be at a reduced speed (unless you had a climbing speed) or it'd break the game. Enemies are in a fortified, elevated position and blocking the ladder up? Cast Spider Climb on your barbarian and send him directly up the wall!

With the new Swarm AI, I can see a fight where you're facing like 30+ goblins attacking e.g., the Druid Grove. Half of them are trying to break down the gates, and the other half are slowly (at quarter speed) but steadily climbing the walls. You're doing your best to prevent them from gaining a foothold; desperately attacking and trying to knock of goblins that are close to the top while also trying to deal with the goblins at the gate.