And even 2 Wizards with no familiars and no friends are MUCH bigger threat than party you described. Thats why amount of people isnt exactly relevant.
It's about a PERCEIVED threat, not the actual threat.
2 skinny dudes wearing cloth robes are much, much smaller of a perceived threat than 10 big marching, great sword wielding hulks in plate armor...even though the 2 dudes in cloth robes are actually 10th level wizards and could wipe out the village in minutes while the 10 hulking warriors are just 1st level fighters that will fall to archer arrows before they can get to the front door.