No, not YOU leave - tell him to leave.
1. Touch the lever
2. imp interrupts
3. talk to him again - and you will see, you have a speech option in your dialogue to inform him about the other merchant meddling with his wares.
4. Imp leaves
5. Touch the lever a 2nd time
