The bored animation happens when you do not move for a while. It goes away as soon as you give a command.

BTW, Baldur's Gate had the characters saying something along with the bored animation.