Do you have any examples of TB games with 5 or 6? They are not what I usually play. I am curious how the combats are in them and if they go faster than here.
I would say maybe 5 people would be a fair compromise, but only if they speed up the enemy turns. Could even make party size yet another menu option, 4 as the standard and a toggle for maybe 6.
Pathfinder: Kingmaker was originally RTwP, but they recently added a turn-based mode derived (with permission) from a mod that added it. It goes relatively quickly, and keeps quite close to the tabletop Pathfinder rules (although flanking is severely simplified).