Well, I'm pretty sure imps are inter-dimensional beings. Pretty much every imp you run across is either in another dimension or has some kind of access to other dimensions.

Except for all those imps in the Wastelands. They may have just been summoned there.

There's also an ancient, deserted imp castle in the Dark Forest. But who knows if they originally built or just lived there at some point...

I think we'll just have to get used to Larian playing fast and loose with the lore. No glaring contradictions. It all kinda fits together.