I was able to avoid it the first time playing. yes they saw me and killed me before I knew they were there, but after a reload I just snuck around them and avoided that section all together. nothing back there beyond a chest I think. Can walk down the other path and jump across to get to the mushroom people and not have to fight the minotaurs