It's almost certainly a factor. A voice can either enhance or detract from a specific character. Not just in terms of the quality of the voice acting, but how often you hear the same voice actor or actress in a role.
I really like Jennifer English and Ben Starr, for example, though at the same time I dislike how present they are due to their success. As talented as they are, it's immersion breaking when the protagonists in multiple games basically end up with the exact same voice that is quite obviously 'them'.
This is also the case with visual acting in movies and TV shows. Certain actors and actresses can be near unrecognisable across different roles in their career, whereas others end up essentially just playing the same character over and over again.
I also think some voice actors and voice actresses do not do themselves any favours by embracing the more parasocial elements of any given fanbase, especially when weighing in on controversial and divisive issues.
I was surprised to find out that Devora Wilde voiced Clea in Expedition 33. Only after knowing this and listening attentively could I recognise some intonations from Lae'zel, but without knowing I'd never have gusessed. She really gets into the skin of the character she's playing.
Jennifer plays well also, but the voice of Maëlle is just ... Shadowheart.
