ritingfool is right. Clicking on your inventory will make sure it gets placed 'in front of' your charm window, it won't actually close it. This can be pretty annoying if you're playing on a low resolution.

Tovi May Raan refresh you!