I've played a couple hours with voice acting and a couple hours without. I'm enjoying the game much more with voice-acting off. I had the same experience when I disabled voice-overs in Pillars of Eternity.
I understand perfectly well that full voice-overs were necessary for mass appeal on the PS4/Xbone, and it was certainly a good idea to have that done for the Enhanced Edition. I'm not sure this game would be taken as seriously by console critics as it has been had Larian not added comprehensive voice-acting. That said, I think the issue that Swen has referred to before--that is, the fine line between what is humorous and what is absurd--is made that much more problematic to deal with when everything is voice-acted.
In my theater of the mind I provide the voices. With one fewer variable (speech), my imagine is freer and I'm able to interpret the script in a way that makes the most sense to me. With voice enabled, I'm strictly an observer relying on a disparate cast of strangers' interpretation of the their little section of a very large script, which is often hit or miss (and unfortunately, I'm able to judge this right away because I've read the text long before the voice-over has ended). This isn't a 90 minute animated film where every phoneme is tightly directed, nor is there a similar budget, so I don't hold this short-coming against Larian. I don't envy the people responsible for directing the voice talent as this must have been a daunting undertaking.
The voice-overs are by no means poor. Most are in fact quite good, but with the tone being set solely by text (along with visual queues to a smaller extent), I'm having a significantly more cohesive experience and the world is so much more real to me.