YES PLEASE! This happens to me so much when using a controller, it's maddening!

Personally, I'd be OK with the long-press being required only when we still have our main action remaining.