Previous studies have shown a reduced lateralization of brain functions in women compared with men. Similarly, some studies have shown that the inter-hemispheric transfer (IHTT) of information is asymmetric in men, with faster latencies in the RH→LH compared with the LH→RH direction, and symmetric in women. The aim of the present study was to investigate IHTT and hemispheric lateralization during face processing in the two sexes. Event-related potentials (ERPs) were recorded in strictly right-handed people (16 men and 17 women) engaged in a face-sex categorization task. Occipital P1 and occipito/temporal N170 were left lateralized in women and bilateral in men. Overall the data suggest a certain involvement of the LH in face feature analysis (possibly related to sex-coding) in both sexes. N170 to contralateral stimuli was larger over the RH in men and the LH in women. IHTT was approximately 4 ms at the P1 level and approximately 8 ms at the N170 level. It was asymmetric in men, with faster latencies in the left visual field (LVF)/RH→LH (170 ms) direction than in the right-visual field (RVF)/LH→RH (185 ms) direction and symmetric in women. These findings suggest that the asymmetry in callosal transfer times might be due to faster transmission times of face-related information via fibers departing from the more efficient to the less efficient hemisphere. Overall, our findings also support the notion that the transfer time of visual inputs might be more rapid and symmetric in women than in men.