Facial emotions express our internal states and are fundamental in social interactions. Here we explore whether the repetition of unattended facial emotions builds up a predictive representation of frequently encountered emotions in the visual system. Participants (n = 24) were presented peripherally with facial stimuli expressing emotions while they performed a visual detection task presented in the center of the visual field. Facial stimuli consisted of four faces of different identity, but expressed the same emotion (happy or fearful). Facial stimuli were presented in blocks of oddball sequence (standard emotion: p = 0.9, deviant emotion: p = 0.1). Event-related potentials (ERPs) to the same emotions were compared when the emotions were deviant and standard, respectively. We found visual mismatch negativity (vMMN) responses to unattended deviant emotions in the 170–360 ms post-stimulus range over bilateral occipito-temporal sites. Our results demonstrate that information about the emotional content of unattended faces presented at the periphery of the visual field is rapidly processed and stored in a predictive memory representation by the visual system. We also found evidence that differential processing of deviant fearful faces starts already at 70–120 ms after stimulus onset. This finding shows a ‘negativity bias’ under unattended conditions. Differential processing of fearful deviants were more pronounced in the right hemisphere in the 195–275 ms and 360–390 ms intervals, whereas processing of happy deviants evoked larger differential response in the left hemisphere in the 360–390 ms range, indicating differential hemispheric specialization for automatic processing of positive and negative affect.