It's probably hard to implement staying halfway on the ladder. So I guess it would be fine if you could climb 20+ meter ladders in a single turn but IMHO then at least your turn should be finished.
Another possibility would be to make climbing ladders a full action. So that you at least lose your attack for one turn.
I mean, maybe? I don't think either you or I really know how difficult it'd be. Obviously it's more complicated than simply pausing a ladder-climbing animation halfway through, but how much so? Idk.
Your
movement (not turn) ending after climbing a too-tall ladder would be a fine solution though. As long as your characters were more than 5 feet from the top of the ladder, they'd then be safe from melee attacks. And/or maybe you're not allowed to climb a ladder if you don't have enough movement, thus possibly requiring you to dash?
I generally don't like modifying the action economy in 5e without good reason, and "we didn't feel like putting in the effort to make climbing cost movement or implement another better solution" isn't a good reason. That said, a full action sounds too costly so a bonus action cost might be better? Since Shove is a BA, this would prevent cheese where you climb a ladder and shove an enemy off.