It sounds like you're saying
"People who don't understand the context will not understand the humor."
Which is kind of a given...