So long as they handle difficulty levels well, then I don't see how it would matter too much. If it's too easy at 6, then bump the difficulty up; if it's too difficult at 4, then bump the difficulty down.