Would a godless paladin truly be non-religious?

PHB pg82. 'Although many paladins are devoted to gods of good, a paladin's power comes as much from a commitment to justice itself as it does from a god.'

Holding justice up so high to the point holy power springs forth sounds pretty much text-book zealotry. And you still need a holy symbol for your spells. And prayer time to prepare them.

Maybe no god makes it not a religion, but it's something real close.