Yes, this is a known issue. Picking up scrolls that way will also trigger the skill (though at least you can right click to cancel).