He is nimble for sure but still though as nails as well.
Just check all the scars he has :p
It's true he was tough enough to survive the extended experimentation after the Trial of Grasses. He's definitely superhuman in most categories, but it's hard to imagine a really slow Geralt. Even the games don't come close to how fast Geralt is in the books. :P