I agree that we should be able to switch between characters. Not only because of the difference in abilities but also because different races and classes have unique options depending on the conversation.
Also, if you use a party member who isn't a PC, the other characters don't react to your choice. I understand the reasoning behind this, but I think it's not fair. If my character has low charisma I might want to use Wyll or someone else in social interactions.