You can not use telekinesis to get the gems. You can avoid the house of madness until after getting your teleporter stones back, but I don't think there is a way to toss one outside.