I think it is a mechanical limitation, but I do agree with you it should be possible to immediately switch to another character and start casting with that one