I find that some of the UI buttons are so small (especially around the quick bar and combat log area) that it's very easy to miss what I'm aiming for and click on the ground or a character instead. Not sure if the OP is having a similar issue, but it might be that some UI fixes will resolve their problem as well.