Even methods that are purposely for pulling objects towards you do not move the gem; there is no in-game way to retrieve it safely at the the moment that I know of, and it doesn't seem like you are intended to be able to do so. Breaking it is one way to stop the trap, however.
