I just started playing Divinity Original Sin: Enhanced Edition on my iMac and have the same issue. After a ton of testing of different settings, I've come to the conclusion that this is a bug.
I've noticed the same thing as others, that you can hover over different parts of the overlay windows and in some positions it will go though to what is underneath. But what I noticed that I haven't seen others mention is that I can move my mouse a tiny fraction to get swap between layers without the cursor moving at all. For example, I can open up the inventory, switch to the skills tab, and then move the mouse a fraction of a pixel and see the highlight flash back and forth between skills and the things below it.
My best guess is there is some sort of rounding error/bit error in how the cursor positions are handled.
My specs:
macOS High Sierra 10.13.5
iMac (Retina 5K, 27-inch, Late 2014)
4 GHz Intel Core i7
32 GB 1600 MHz DDR3
AMD Radeon R9 M295X 4096 MB
