They don't "use hitboxes" but they check for line of sight, which is a slightly different concept.
Past that, if it's a hit or miss is strongly dependent on what type of attack is used (i.e. magic missiles always hit if there's LOS while a firebolt needs to pass a roll).
