I won't take the credit for figuring it out. Forumer tonyb pointed me in the right direction.

I think those pressure plates were actually meant to be traps I guess?