Lowering the goblins hp would make the fights a little too easy in my opinion. Let the gobbos keep their hp buff. I prefer to not have my enemies killable in one hit.
You can always increase their numbers. That's sort of the point why Wizards and CC is a thing. Different dynamics than everyone just attacking one goblin at a time.
