Yeah, I find that annoying as well. I don't see why the game has to lock onto the character casting the spell when out of combat.