I got recked in driftwood, although that was more because I didn't go the correct way. All of fort joy is more or less a cake walk. I didn't even have difficulty with fort joy with a 4 person team of mixed damage on classic, what party setup are you running? it must have been awful. That's the only way I could see someone struggling so much with fort joy that they need to turn it down to easy. The only fight I felt remotely challenged in fort joy was the crocodiles, but only if I did them at lvl 2 instead of 3. Once you hit 3 they get significantly easier.
So I really have no idea where this complaint comes from as it's the exact opposite of my experience.