A CR and a lvl are not the same though.
A CR3 creature alone would be a challenge for a lvl 3 party. Mind you circumstances like high ground or suprise can play a very big role in succes or failure.
That's a common misconception, and the reason why I vastly prefer using XP, rather than CR when designing encounters.
What you are saying is true, although challenge, should rather be a "challenge". CR 3 is a medium level encounter for 3rd level party, party can succeed in iirc 7 medium encounters per rest, meaning that encounter is challenging when you've literally burnt through all your spells, and are on your last legs. Names of the encounter tiers are misleading as well. Trivial, easy, and medium are trivial for rested party. Hard is what you would expect from easy-medium encounter for a rested party. Deadly encounter, now we are talking. A fully rested party can take about two deadly encounters, meaning you can easily strech it to double, even triple the XP treshold of a deadly encounter assuming the party is optimised, and they have the advantage in the form of surprise, and high ground.
With that in mind, the 4 members big party of level 3, can actually beat the CR 6, maybe 7. An example of those CRs are white, green or black dragons (young dragons), or a mindflayer :p
