I always just felt as though "you failed your action, and thus wasted your turn achieving nothing" was punishment enough for the failure, personally.