Experience is determined by the "gain" field in characters.txt and scaled by act part.

Try changing:
data "Gain" "5"

to

data "Gain" "None"

for enemies you don't want to give experience. Note that a gain of 0 is different than a gain of None.