It's almost certainly a factor. A voice can either enhance or detract from a specific character. Not just in terms of the quality of the voice acting, but how often you hear the same voice actor or actress in a role.
I really like Jennifer English and Ben Starr, for example, though at the same time I dislike how present they are due to their success. As talented as they are, it's immersion breaking when the protagonists in multiple games basically end up with the exact same voice that is quite obviously 'them'.
This is also the case with visual acting in movies and TV shows. Certain actors and actresses can be near unrecognisable across different roles in their career, whereas others end up essentially just playing the same character over and over again.
I also think some voice actors and voice actresses do not do themselves any favours by embracing the more parasocial elements of any given fanbase, especially when weighing in on controversial and divisive issues.
