Social Cognitive and Affective Neuroscience Advance Access published online on August 24, 2007
Social Cognitive and Affective Neuroscience, doi:10.1093/scan/nsm026
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Happy and fearful emotion in cues and targets modulate event-related potential indices of gaze-directed attentional orienting
1Department of Psychology and 2Department of Psychiatry, Yale University, New Haven, CT 06510, 3Department of Psychology, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599 4Department of Psychology, Texas State University at San Marcos, San Marcos, TX 78666, USA and, 5Center for Cognitive Neuroscience and Department of Psychology and Neuroscience, Duke University, Durham, NC 27708
Correspondence should be addressed to Kevin S. LaBar, PhD, Center for Cognitive Neuroscience, Box 90999 Duke University, Durham, NC 27708, USA. E-mail: klabar{at}duke.edu
 |
ABSTRACT |
|---|
 TOP ABSTRACT METHODSRESULTSDISCUSSIONCONCLUSIONSConflict of InterestACKNOWLEDGEMENTSREFERENCES |
|---|
The goal of the present study was to characterize the effects of valence in facial cues and object targets on event-related potential (ERPs) indices of gaze-directed orienting. Participants were shown faces at fixation that concurrently displayed dynamic gaze shifts and expression changes from neutral to fearful or happy emotions. Emotionally-salient target objects subsequently appeared in the periphery and were spatially congruent or incongruent with the gaze direction. ERPs were time-locked to target presentation. Three sequential ERP components were modulated by happy emotion, indicating a progression from an expression effect to a gaze-by-expression interaction to a target emotion effect. These effects included larger P1 amplitude over contralateral occipital sites for targets following happy faces, larger centrally distributed N1 amplitude for targets following happy faces with leftward gaze, and faster P3 latency for positive targets. In addition, parietally distributed P3 amplitude was reduced for validly cued targets following fearful expressions. Results are consistent with accounts of attentional broadening and motivational approach by happy emotion, and facilitation of spatially directed attention in the presence of fearful cues. The findings have implications for understanding how socioemotional signals in faces interact with each other and with emotional features of objects in the environment to alter attentional processes.
Keywords: facial affect; positive psychology; evoked potentials; shared attention; social neuroscience
Gaze direction and emotional expression are used in social contexts to direct visuospatial attention to salient features of the environment. The attentional focus and motivational state of a social partner can be discerned in part by interpreting his or her communicative facial cues. These socioemotional signals help in making inferences about the significance of stimuli in the environment, including those that are outside of one's own field of view. Attentional focus is most readily inferred by noticing the direction of a partner's gaze, although other cues such as body and vocal gestures are also used in social communication. Emotional facial expressions provide information about the motivational state of the actor (Ekman and Oster, 1979) as well as changes in one's level of safety and threat.
People rely on their understanding of changes in facial behaviors and gaze direction in social situations to respond appropriately to both a partner as well as to the stimulus that elicited his or her change in affect and attentional deployment. For example, children learn to use emotional expression and attention-directing cues displayed by a parent to determine whether a new person or object is safe to approach, a phenomenon called social referencing (Klinnert et al., 1983). In this way, a change in motivational state of the child is initiated by assessing the emotional nature of the situation as revealed by a parent's facial and body cues.
Withdrawal-oriented motivational states are activated during conditions of impending threat whereas approach-oriented motivational states emerge under circumstances leading to a potential reward (Davidson, 1998). In social settings, the display of fear on another person's face is a cue that an impending threat is present in the environment, and a happy expression represents a situation where a reward could be expected. Approach states are associated with greater likelihood to interact with environmental stimuli whereas withdrawal states lead to environmental disengagement (Sobotka et al., 1992; Hillman et al., 2004). Thus, the combination of multiple social signals from a partner permits inferences regarding the salience and safety of events in the environment and can determine changes in attention, action and motivational state during social communication.
Because objects are foveated for visual acuity, gaze direction generally provides more precise information than other bodily cues regarding the spatial localization of one's attentional focus, except in cases of deception or covert orienting. However, gaze information alone is ambiguous with respect to signaling the emotional relevance of the object of a partner's attentional focus. Gaze following may have evolved to identify the location of environmental objects that have both positive and negative consequences, such as a potential food source or the sudden emergence of a predator. Therefore, gaze information must be combined with that gleaned from facial expression, vocal affect or bodily gestures to react in an advantageous manner to observed changes in a partner's gaze direction.
Effects of gaze and emotion on attention have been experimentally investigated in studies that vary the standard Posner attentional cuing paradigm by using faces as attention-directing cues (Friesen and Kingstone, 1998). In this task, the peripherally or centrally presented symbolic cues are replaced with a socially relevant cue, a face. At the start of each trial a face with direct gaze is presented at fixation, after a short interval the pupils shift to represent leftward gaze or rightward gaze, or hold position for a direct gaze. Targets are randomly presented at a location in the periphery of the left or right visual field. Participants are faster at responding to targets (detection, localization, or identification) when presented at the gazed-at location (valid) compared to the opposite visual field (invalid) or when there was a direct gaze.
Using this approach, only a few behavioral studies have directly investigated the relationship between emotional expression and gaze during attentional target detection or identification tasks, and those that have been conducted provide mixed results (Hietanen and Leppanen, 2003; Hori et al., 2005; Graham et al., 2006; Putman et al., 2006; Tipples, 2006). Our previous studies (Graham et al., 2006) have shown that targets are responded to more quickly when observing faces that express a change in emotional expression (fearful, happy or disgust) regardless of the match between gaze direction and target location. However, some studies have shown that facial expression can interact with gaze validity, with larger validity effects in the presence of a fearful face (Putman et al., 2006; Tipples, 2006) or a happy face (Hori et al., 2005; Putman et al., 2006) relative to a neutral face. Other studies have only shown this effect in socially anxious participants (e.g. Matthews et al., 2003). Another group of studies have found no effect of facial expression (fearful, angry, disgust or happy) and no interactions between gaze direction and facial expression on attention (Bayliss et al., in press; Hietanen and Leppanen, 2003). Currently, it is unclear which methodological factors are contributing to these discrepancies.
Event-related potentials (ERPs) have been used to complement behavioral methods by directly measuring the impact of social cues on the neural processing of targets. ERPs permit analysis of spatiotemporal dynamics of brain activity and therefore can provide additional insights into component processes because they do not rely upon convergence of effects on a single output measure, such as reaction time. Previous ERP studies of gaze-directed attentional orienting have focused on a sequence of ERP waveform components called the P1 (at occipital electrode sites between 70–100 ms post target onset), N1 (at parietal/occipital electrode sites 
150–200 ms post target onset) and P3 (at central/parietal/midline electrode sites 300–500 ms post target onset). The occipital P1 component is thought to represent the initial processing of a stimulus in extrastriate cortex, while the parietal/occipital N1 is thought to index target discrimination processes. Modulation of the N1 component is generally seen during tasks which require a choice compared to a simple response (Mangun and Hillyard, 1991). The P3 is a complex of related components, and the parietal P3 investigated here (sometimes called P3b) has been shown to serve as an index of location expectancy in spatial attention tasks (e.g. Mangun and Hillyard, 1991; Schuller and Rossion, 2001, 2004, 2005), as well as an index of object categorization processes (Donchin, 1979).
Schuller and Rossion (2001, 2004, 2005) demonstrated enhanced P1 amplitude in response to gazed-at targets (validly-cued) compared to targets presented at a location in the opposite hemifield from the gazed-at location (invalidly-cued), as well as greater P3 amplitude in response to invalid compared to valid targets (Schuller and Rossion, 2001, 2004, 2005). However, these studies did not vary the expression on the face. In real-world contexts, attentional orienting to salient targets is likely driven by both bottom-up influences of target valence and top-down (expectancy) influences of cue expression. Whether these two sources of emotion operate independently or are integrated during target processing is unknown.
In the current study, a modified gaze-cuing paradigm was used in which dynamic gaze shifts and dynamic expression changes were incorporated in face cues on every trial (neutral to fearful, or neutral to happy) and attentional targets were objects that varied in emotional meaning (a baby and a snake). Changes in gaze shifts, cue emotion and target emotion were fully crossed. Thus, participants were presented with a complex attention-directing face cue that had the appearance of emotionally reacting (happy or fearful expressions) just prior to the appearance of either a positive or negative target.
In a preliminary study, we incorporated a dynamic change in fearful expression along with a dynamic gaze shift and demonstrated separable effects of facial expression and gaze-direction validity on attentional orienting to emotionally neutral targets (Fichtenholtz et al., 2007). Effects of facial expression were seen early in target processing (fearful > neutral, P1 component), while an effect of gaze-direction validity was seen on the P3 component, consistent with Schuller and Rossion (2001, 2004, 2005). Given these and related behavioral findings, in the present study we expected to observe independent effects of facial emotion and gaze direction validity in the reaction time (RT) data. Previous ERP experiments using gaze-cuing paradigms suggest that enhanced early processing should occur for targets presented at the gazed-at location. However, the addition of a facial expression change to a gaze-directing cue minimizes the initial attentional impact of gaze direction (Fichtenholtz et al., 2007), suggesting that only effects of facial expression (and not gaze validity) should be seen early during target processing (P1 and N1 component).
Changes in facial expression that predict emotional outcomes should create a situation where the participants are in an approach-oriented anticipatory state following the happy expression and a withdrawal-oriented anticipatory state following the fearful expression. This change in motivational state could affect all succeeding target stimuli regardless of location. It has been shown that negative emotions cause a narrowing of attentional focus while positive emotions engender attentional broadening (Kienen, 1987; Conway and Giannopoulos, 1993; Derryberry and Tucker, 1994; Basso et al., 1996; Isen, 1999; Maxwell et al., 2005). This valence effect has been demonstrated by equivalent performance for both left and right visual field targets for happy expressions, whereas performance varied by visual field for angry and neutral expressions. In this situation, enhanced processing of target stimuli following a happy face would not be limited to those targets presented at the gazed-at location. Therefore, we hypothesized that gaze-directed attentional effects would occur later during target processing (i.e. P3 modulation) and would be more prominent for fearful emotion. Due to the dynamic nature of the gaze and expression changes in the cue stimuli, it is difficult to differentiate the responses to the gaze-shift and the expression change. Consequently, the analysis presented here is focused on target processing.
|
METHODS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSConflict of InterestACKNOWLEDGEMENTSREFERENCES |
|---|
Participants
Twenty healthy right-handed young adults (M 20.15-years-old ± s.d. 1.90; 9 males) provided written consent to volunteer in this experiment. All of the participants were Duke University students and received either course credit for their participation or were compensated at a rate of $10 per hour. Participants were screened for a history of neurological and psychiatric disorders, substance abuse and current psychotropic medications. Due to the fear-relevant nature of one of the target stimuli, each participant completed the snake anxiety questionnaire (SNAQ; Klorman et al., 1974). All participants scored at least 1 s.d. below the mean for phobic subjects (Fredrikson, 1983). Therefore, no participants were excluded from the analysis due to their scores on the SNAQ. One subject did not complete the task due to technical difficulties, and data from three participants were excluded because of excessive artifacts. Analysis was conducted on the remaining 16 participants (9 females, 7 males), who had a mean age of 20.5 (s.d. = 1.93). The Duke University Medical Center Institutional Review Board approved the protocol for this study.
Task parameters
The task consisted of nine runs. Each run contained 96 trials, four each of 24 stimulus categories, and lasted 5.6 min. Trials were pseudorandomized in an event-related design across participants. Across all nine runs, participants saw a total of 36 trials for each category. Each trial consisted of a dynamic facial cue stimulus that changed both gaze direction (left, right) and facial expression (happy, fearful), followed by presentation of a target object (baby, snake). All of these variables were fully crossed in the experimental design. The nomenclature for the direction of gaze is based upon the participants’ frame of reference so on a leftward gaze shift trial the pupils move to the left of center. The eye gaze cue consisted of three phases. The first phase was the presentation of a neutral face for 300 ms. The second phase consisted of the presentation of a 50% fearful or 50% happy expression and a left or right eye gaze presented for 50 ms. The third phase consisted of the presentation of a 100% fearful or 100% happy expression and a left or right eye gaze presented for 50 ms. Following this third phase, the cue stimulus maintained its facial configuration and remained on the screen for the remainder of the trial (1100 ms). Spatial location of the target was validly cued by gaze shifts 50% of the time, and congruency of emotion in cues and targets (e.g. babies following happy faces) occurred 50% of the time.
The dynamic expression change created a situation where the participant saw the cue stimulus become either afraid or pleased in response to the upcoming stimulus. Additionally, previous research has shown that emotional expressions are more accurately identified when presented dynamically (Ambadar et al., 2005) which should enhance the perceived emotional experience of the participant. One hundred milliseconds after the onset of the gaze shift, the attentional target was presented for 100 ms. The attentional target consisted of a rectangular image of either a snake or a baby presented in the periphery (2.1° above fixation, 7.4° left or right of fixation) of the upper left or right visual field (see Figure 1 for a visual depiction of the task). One-third of the trials consisted of the cue sequence being presented with no target (catch trials). There was a random inter-trial interval between 1500 and 2000 ms.
|
Throughout the entire run, participants were asked to fixate on a centrally presented cross. The participants’ task was to identify the content (baby or snake) of the target image using two buttons on a game controller. Responses were made with the index finger of each hand. Response mapping was balanced across subjects.
Stimuli
One male actor (P.E.) was selected from the Ekman and Friesen (1978) pictures of facial affect to act as the centrally presented cuing stimulus. One actor was used and compared across emotion categories because previous studies have established that facial identity modulates the perception of facial expression (e.g. Schweinberger and Soukup, 1998). The original photos posed fearful, happy and neutral facial expressions with direct gaze. Adobe Photoshop (Adobe Systems Incorporated, San Jose, CA) was used to manipulate gaze direction so that irises were averted between 0.37° and 0.4° from the centrally positioned irises in the faces with direct gaze. Thus, five digitized grayscale photographs were used—a neutral face with direct gaze as the initial anchor for the morph and the factorial combination of two facial expressions (fear, happy) and two gaze directions (left, right) from the same actor. In order to create realistic dynamic emotional expressions, fearful and happy facial expressions of intermediate intensity were created using the morphing methods outlined in LaBar and colleagues (2003) using MorphMan 2000 software (STOIK, Moscow, Russia). Three morphs depicting 55% fearful or happy/45% neutral expression were created with left- and right-looking gaze and were used to create a more natural-looking appearance of apparent motion. The grayscale facial cuing stimuli were presented at fixation and subtended 6.3° of horizontal and 8.9° of vertical visual angle.
Target stimuli consisted of two photos that were chosen from the IAPS affective picture set (Lang et al., 2001). One stimulus was positively-valent and depicted the face of a baby (image 2070); the other was negatively valent and depicted the face of a snake (image 1120). The stimuli were chosen, not only for their opposing valence, but also because they both shared similar features (i.e. both were pictures of faces with open mouths). The original IAPS photos were cropped to include only the face area and were converted into gray scale photos. The contrast and luminance of the target photos were equated to that of the facial cues. Targets measured approximately 2.5° of visual angle.
Behavioral data analysis
Mean RT and accuracy were computed for each condition and participant. Any trials with RTs shorter than 100 ms or longer than 1000 ms (duration of the face on the screen after target presentation) were excluded from this and all subsequent analyses. Two (Expression: fearful, happy) x 2 (Target Emotion: positive, negative) x 2 (Gaze Direction: left, right) x 2 (Target Location: left, right) ANOVAs were conducted for RT and accuracy separately.
ERP recordings
The electroencephalogram (EEG) was recorded from 64 electrodes in a custom elastic cap (Electro-Cap International, Inc., Eaton, OH) and referenced to the right mastoid during recording. Electrode impedances were maintained below 2 k
for the mastoids, below 10 k for the facial electrodes, and below 5 k for all the remaining electrodes. Horizontal eye movements were monitored by two electrodes at the outer canthi of the eyes, and vertical eye movements and eye blinks were detected by two electrodes placed below the orbital ridge of each eye. The 64 electrodes were recorded with a bandpass filter of 0.01–100 Hz and a gain of 1000, and the raw signal was continuously digitized at a sampling rate of 500 Hz. Recordings took place in an electrically shielded, sound-attenuated chamber.
Because we recorded from 64 electrode sites, the nomenclature used to describe ERP results is based on the International 10–20 system (Jasper, 1958) but with additional information that reflects the increased spatial coverage. Electrodes are identified by 10–20 positions, modified with letters or symbols denoting the following: a = slightly anterior placement relative to the original 10–20 position, s = superior placement, i = inferior placement, and ‘ = placement within 1 cm of 10–20 position.
ERP data reduction
Artifact rejection was performed off-line by discarding epochs of the EEG that revealed eye movements, eye blinks, excessive muscle-related potentials or drifts. For the 16 participants included in the final analysis 13.83% of trials were excluded due to artifacts. The analysis was focused on examining how ERPs elicited by targets varied as a function of cue emotion, target emotion, gaze direction and target location. Thus, averages were calculated for 16 bins formed by crossing two facial expressions (fearful, happy) with two target emotions (positive, negative), two gaze directions (left, right), and two target locations (left or right visual field). Each average was calculated from 250 ms before to 1000 ms after picture onset and digitally low-pass filtered at 60 Hz. By randomizing the different trial types, the impact of response overlap from previous stimuli in the sequence was minimized. Importantly, in order to minimize the differences in target processing due to the physical differences in the cue stimuli, the responses from the cue-only trials were subtracted from the cue-target trials for each trial type separately to isolate the target-related responses (i.e. Fear/Left gaze/Left target trials minus Fear/Left gaze/No target trials). After subtracting the cue-only trials, all channels were re-referenced to the algebraic average of the two mastoid electrodes. The ERP averages for individual participants were then combined into group averages across all participants.
ERP data analysis
Group-averaged ERPs elicited by left visual field (LVF) and right visual field (RVF) targets (collapsed across cue type) were plotted. These plots revealed the three components that were predicted, (i) a positive deflection (P1 component, P130) at occipital electrode sites, peaking at approximately 130 ms, (ii) a negative deflection (N1 component, N180) at central electrode sites, peaking at approximately 180 ms and (iii) a positive-going wave at central electrode sites, extending from 250–750 ms (P3 complex). In order to capture the peak amplitude of each deflection seen in the group-averaged waveforms (Figure 2A) three latency intervals were selected for analyses: 115–145 ms, 165–195 ms and 250–750 ms. Additionally, electrode sites were selected to capture the spatial extent of each component at its peak (see Figure 2B for topography of all components). Mean amplitude during each latency window at each electrode site was entered into further analyses.
|
The ANOVA for the contralateral ERPs evoked by the target stimuli during the time range of the P130 component (115–145 ms) included five factors: expression (fearful, neutral), target emotion (positive, negative), gaze direction (left, right), target location (LVF, RVF) and electrode location (lateral, medial; sites P3i, P4i, O1’, O2’). The ANOVA for the midline ERPs evoked by the target stimuli during the time range of the N180 component (165–195 ms) included the same five factors with the exception that electrode location was coded along the rostral-caudal axis (sites Cza, Cz, Pzs). Unlike the P130 and N180 components, which have distinct peaks, the P3 complex was a broad waveform lasting for approximately 250–750 ms. To aid in the analysis of this component, the time window was broken down into five consecutive 100 ms sections, and a factor of time was added to the same analysis conducted for the N180 component (sites Pzs, Pzi, Ozs). Follow-up comparisons of ERP effects were conducted using additional ANOVAs or Bonferroni-corrected paired-samples t-tests. Greenhouse-Geisser corrections were used where appropriate.
|
RESULTS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSConflict of InterestACKNOWLEDGEMENTSREFERENCES |
|---|
Behavior
Accuracy. The ANOVA on accuracy data revealed significant interactions between expression and target emotion, F(1, 15) = 5.63, P < 0.03, and between gaze direction and target location, F(1, 15) = 5.93, P < 0.03. Regarding the interaction of expression and target emotion, participants were more accurate at identifying the type of target presented when it was emotionally congruent (95.5% correct ± s.d. 0.02) with the facial expression presented in the cue, compared to when the target emotion was incongruent (94.7% correct ± s.d. 0.03). Regarding the interaction between gaze direction and target location, target identification was easier when the target was presented at the gazed-at location (95.6% correct ± s.d. 0.03) compared to the opposite location (94.7% correct ± s.d. 0.03). Because accuracy was near ceiling level of performance, these results should be interpreted with caution.
Reaction time. The ANOVA on RT data revealed a main effect of target location, F(1, 15) = 6.49, P < 0.02, as well as interactions between gaze direction and target location, F(1, 15) = 8.57, P < 0.01, and expression and target location, F(1, 15) = 4.70, P < 0.05. The effect of target location was due to participants responding faster to targets presented in the RVF (RVF: M = 476.64 ± 56.88 s.d.; LVF: M 483.74 = ± 58.58 s.d.). Participants had shorter RTs to targets presented at the gazed-at location compared to the opposite location, consistent with an attentional validity effect (Figure 3A). Not only were participants faster when responding to RVF targets, but also were significantly faster to RVF targets following a fearful compared to a happy face, F(1, 15) = 6.19, P < 0.03 (Figure 3B).
|
Event-related potentials
P130 Component. The ANOVA at contralateral electrode sites revealed a main effect of expression, F(1, 15) = 30.37, P < 0.01, with greater P130 amplitude following happy compared to fearful faces (as seen in Figure 4). A similar effect was seen in the response recorded at ipsilateral electrodes, F(1, 15) = 5.39, P < 0.04, with larger responses to targets following happy expressions.
|
N180 component. The ANOVA revealed significant interactions between expression and gaze direction, F(1, 15) = 4.73, P < 0.05, and expression and target location, F(1, 15) = 6.29, P < 0.02. Analysis of the first interaction showed an effect of expression (happy > fearful) on trials with a leftward gaze, F(1, 15) = 9.81, P < 0.01 (Figure 5). There was no effect on trials with a rightward gaze. Follow-up ANOVAs of the expression x target location interaction revealed a trend towards an effect of expression for targets presented in the LVF, F(1, 15) = 4.51, P < 0.06, with targets following happy expressions showing a larger N1.
|
P3 complex. The ANOVA showed significant main effects of time, F(4, 60) = 10.93, P < 0.01, and expression, F(1, 15) = 7.67, P < 0.02. The main effect of expression showed greater amplitude in response to targets preceded by happy faces compared to fearful faces (Figure 6). Follow-up analyses of the time effect showed that the amplitude increased from 250 to 550 ms and then decreased from 550 to 750 ms (see Table 1 for individual comparisons), following the mean rise and decay of the component amplitude.
|
|
In addition to these main effects, three significant interactions were found: expression x gaze direction x target location, F(1, 15) = 9.47, P < 0.01; time x target emotion, F(4, 60) = 12.84, P < 0.01; and time x expression x target location, F(4,60) = 3.14, P < 0.03. Analysis of the first interaction revealed that, in the presence of a fearful face, amplitude of the P3 complex was smaller in response to targets presented at the gazed-at location (validly cued) compared to the opposite location (invalidly cued), reflecting an increase in the processing of invalidly cued targets, F(1, 15) = 4.79, P < 0.05. No validity effect was seen for happy expression trials, and these trials had a mean value that was similar to the invalidly cued fearful expression trials (Figure 7).
|
The interactions including the factor of time were further investigated using follow-up ANOVAs. Early during the P3 complex time window (250–550 ms), positively valent targets (baby) had greater amplitude than negatively valent targets (snake). This effect was reversed later during the P3 complex time window (550–650 ms). While the overall amplitude was similar as a function of target valence, the time interaction indicates that positive targets were processed faster (Figure 8). A peak latency analysis confirmed this result; a main effect of target emotion, F(1, 15) = 14.85, P < 0.01, showed that the P3 peaked earlier to positively valent targets.
|
The time x expression x target location interaction was also explored using a peak latency analysis and revealed a significant expression x target location interaction, F(1, 15) = 10.45, P < 0.01. Further analysis of this interaction revealed that happy-cued targets (454.18 ms ± s.d. 93.04) were processed faster than fear-cued targets (492.39 ms ± s.d. 100.68) presented in the RVF, F(1, 15) = 19.41, P < 0.01, but no difference between happy-cued and fear-cued trials in the LVF (Happy: 464.22 ms ± s.d. 94.92; Fear: 472.29 ms ± s.d. 106.16) was found.
|
DISCUSSION |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSConflict of InterestACKNOWLEDGEMENTSREFERENCES |
|---|
The current study used ERPs to characterize the stages of processing at which gaze-directed attentional orienting was modulated by emotional information in facial cues and target objects. Results demonstrated emotional effects across a sequence of waveform components (P130, N180, P3 Complex), elicited by target stimuli that have been consistently implicated in gaze-directed cuing studies (Schuller and Rossion, 2001, 2004, 2005). Early stages of processing were modulated by happy emotion. P130 amplitude, maximal over contralateral occipital sites, was greatest in response to targets following happy faces (main effect of cue emotion). Subsequently, N180 amplitude, which had a broad central distribution, was greater for targets following a happy face with a leftward gaze relative to the other conditions (cue emotion x gaze direction interaction); this interaction with gaze direction is discussed in more detail subsequently. The valence of the target stimuli (i.e. baby vs snake) did not affect target processing until the P3 complex, and was characterized by reduced peak latency to the positive target (baby), particularly in the RVF. Fearful expression reduced parietally distributed P3 amplitude for gazed-at targets (cue emotion x gaze validity), providing evidence for the presence of spatially directed attention. Behavioral results validated that attention was effectively manipulated by the gaze-cuing paradigm.
The current study builds upon prior research by examining the impact of concurrent changes in cue and target emotion and their interactions with gaze shifts during attentional orienting. The results demonstrate dissociable attentional and timing effects of cue and target emotion, as well as interactions between facial expression and gaze on ERP components elicited by an attentional target. The ERP effects unfolded in a temporal progression from expression effects to expression x gaze direction interactions and finally expression x gaze validity interactions. The majority of effects were driven by happy emotion, with an important exception—the facilitation of spatially directed attention on the posterior P3 complex by fearful expression. Because the early happy effects of cues and targets were not modulated by gaze validity, we interpret these findings as implicating a decrease in the special selectivity of attention that accompanies approach-oriented motivational orientation. In contrast, fearful expression specifically facilitated spatially directed attention by gaze cuing. Modulation of each ERP component is discussed in turn below.
P130 component
The facial expression effect on the target-related P130 component is consistent with the idea that the presence of the happy expression induced an approach-oriented motivational state, which increased early processing of all subsequent targets regardless of location. Because P130 modulation was the earliest effect observed, the finding supports the idea that facial expression is a more relevant social signal than eye gaze for initiating shared (joint) attention in dyadic encounters. The absence of a gaze-directed validity effect in this time window, which was reported by Schuller and Rossion (2001, 2004, 2005), is likely due to the importance of the change in facial expression, which was not present in their experimental design. The sudden emergence of a happy facial expression in the context of uncertain emotional outcomes may have caused a shift from a neutral to an approach-oriented motivational state, since happy expression signaled from a partner sets up an expectation that a potentially rewarding stimulus has just entered the environment. Consequently, more attentional resources are made available to engage the potential rewarding stimulus regardless of target location. Follow-up studies should be conducted using multiple face exemplars and other positive expressions of emotion to ensure that the effects generalize.
N180 component
The facial expression x gaze direction interaction on the target-related N180 component demonstrates a temporal shift from the initial influence of expression on target processing to the later combined influence of both social cues. In the absence of an expression manipulation, previous studies (Schuller and Rossion, 2001, 2004, 2005) have demonstrated significant gaze validity effects on this component. The current study demonstrated enhanced processing of targets following a leftward gaze and a happy facial cue, and targets presented in the LVF showed enhanced processing following a happy facial cue. When taken together, these results suggest that the presence of a fearful facial stimulus recruits more attentional resources from the right hemisphere than a happy facial stimulus. When the right hemisphere attentional mechanisms are engaged, either by a leftward cue or the presentation of a target to the LVF, greater attentional resources are available to engage in processing the target if the preceding facial cue was happy. This effect may be related to our behavioral finding that participants respond faster to targets presented in the RVF if they were preceded by a fearful facial cue, which suggests that the left hemisphere is engaged in processing the happy facial cue to a greater extent than the fearful cue. Thus, the processing resources available to each hemisphere may be preferentially engaged by the motivational significance of the facial cues (approach: left hemisphere, withdrawal: right hemisphere), leaving fewer resources available to process the upcoming target stimuli.
The gaze by emotion interaction may also reflect differential processing of facial expression dependent upon gaze direction, as suggested by prior behavioral and functional neuroimaging research (Adams and Kleck, 2003, 2005). Although not an effect of gaze direction validity, the differential effect of expression by gaze direction demonstrates that the information present in the gaze shift is influencing target processing at this stage. We note that the spatial distribution of the N180 component seen in the current data (central/parietal focus over the midline scalp; Figure 2B) is different from the lateral-occipital distribution of the N1 component seen during many studies of attentional orienting (Luck et al., 2000). This distribution is more consistent with the anterior N1, which is usually seen earlier (100–150 ms) after target onset (Luck, 2005), and which is also thought to index target discrimination processes.
P3 Complex
The decreased latency of the posterior P3 complex response to targets following happy compared to fearful expressions in the RVF is consistent with theories of left hemispheric specialization for approach motivational states (Davidson, 1998). Although the spatial distribution of the P3 complex did not differ following happy and fearful expressions, the organization of the visual system dictates that a stimulus presented in the RVF is initially processed by the left hemisphere, which may be primed to engage in upcoming stimuli during an approach-oriented motivational state (Davidson, 1998). The effect of target valence on P3 complex latency suggests that less time is required to integrate a positively than a negatively valent image into the current behavioral context, consistent with previous ERP studies that have demonstrated longer P3 latency for negative images in a target discrimination task (Akamine and Kida, 2001).
The presence of the expression by gaze validity interaction on the posterior P3 complex shows a spatially directed attentional benefit for fearful expression. Previous investigations of target processing in response to gaze shifts have shown a similar effect (Schuller and Rossion, 2001, 2005). Although validly and invalidly gazed-at targets occur in equal proportion, the gaze cue may nonetheless reflexively increase expectation for the impending target location. Because P3 amplitude indexes the violation of the location expectation, its reduction in the presence of a gazed-at location in the presence of a fearful expression suggests a facilitation of contextual updating mechanisms (Mangun and Hillyard, 1991), consistent with previous studies of negative facial emotion and attentional orienting (Mogg and Bradley, 1999; Mogg et al., 2000; Holmes et al., 2003). Despite its breadth, we did not observe any shifts in topography along the peak and plateau regions of the P3 complex response, and we found no evidence of a more frontally distributed P3a component. Nonetheless, future studies could use principal components analysis or other techniques to determine whether there are selective gaze and emotion effects on different components within the P3 complex.
Role of affective priming
Studies of affective priming have shown that behavioral responses to an emotionally valenced target is facilitated when preceded by an emotionally congruent stimulus relative to a neutral stimulus or one with an emotionally incongruent valence (reviewed in Fazio, 2001). Given that cues and targets in the present attentional orienting study were fully crossed in emotional valence and presented in rapid succession, it may be surprising that no emotional congruency effects were seen in the ERP or RT data. Although participants were behaviorally more accurate on emotionally congruent trials (e.g., fearful face–snake target), accuracy levels were close to ceiling, suggesting that the discrimination task was easy to perform. In addition to task difficulty, several aspects of the experimental design may have minimized affective priming effects. For instance, when non-evaluative tasks are used, affective priming is minimized (de Houwer et al., 2002). Additionally, Musch and Klauer (2001) demonstrated that when an emotionally salient target appears at a predicted location, the affective priming effect is eliminated. Therefore, the effectiveness of the gaze direction validity manipulation likely reduced the potential for affective priming effects in the current study. In addition, the task did not explicitly ask participants to assess the emotional meaning of the target stimuli (such as its arousal or valence levels), only to categorize the subject matter (i.e. baby or snake). Finally, because the same stimuli were repeated throughout the study, it is possible that affective priming effects decreased over the course of the experiment. Because the current study is the first to manipulate emotion in both cues and targets during gaze-directed attentional orienting, further research is warranted to identify the conditions under which affective priming effects are seen.
|
CONCLUSIONS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSConflict of InterestACKNOWLEDGEMENTSREFERENCES |
|---|
We compared the effects of multiple facial expressions on the processing of peripherally presented emotionally salient targets during gaze-directed attentional orienting. The results are novel both in demonstrating the temporal staging of cue and target effects as well as in dissociating the impact of happy and fearful emotion on attentional orienting. Early processing benefits were found for facial expression cuing whereas emotional target effects emerged later, when target identity is integrated into the socioemotional context set up by the expectancy manipulation. The majority of effects were driven by positive emotion but were not spatially directed, consistent with the idea that happy expressions in social exchanges set up global expectations for possible rewarding outcomes that induce approach-oriented motivational states and broaden attentional focus. In contrast, fearful expressions facilitated the spatial direction of attention cued by eye gaze, in accordance with its social role in communicating the detection of a specific threat in the local environment. In conclusion, the deployment of attention by observing multiple dynamic facial signals in others emerges over sequential processing stages and is distinguished from bottom-up effects driven by the emotional significance of environmental stimuli. The ERP results reveal the mental chronometry of shared (joint) attention across emotions that differ in their communicative functions for revealing salient features of the environment to others. Our findings advance a neuroscientific understanding of how inferences are made from observing changes in several facial cues during social exchanges to direct attention in settings where emotional consequences cannot be forecasted.
|
Conflict of Interest |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSConflict of InterestACKNOWLEDGEMENTSREFERENCES |
|---|
None declared.
|
ACKNOWLEDGEMENTS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSConflict of InterestACKNOWLEDGEMENTSREFERENCES |
|---|
This research was supported by a Ruth L. Kirschstein Individual NRSA Fellowship (F31 MH074293) to H.M.F., a National Institutes of Health grant (R01 MH66034) to J.B.H., and a National Science Foundation CAREER award (0239614) and a National Institutes of Health grant (R01 DA14094) to K.S.L.
Received May 9, 2007. Accepted May 14, 2007.
|
REFERENCES |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSConflict of InterestACKNOWLEDGEMENTSREFERENCES |
|---|
Adams RB, Kleck RE. Percieved gaze direction and the processing of facial displays of emotion. Psychological Science (2003) 14:644–7.[CrossRef][Web of Science][Medline]
Adams RB, Kleck RE. Effects of direct and averted gaze on the perception of facially communicated emotion. Emotion (2005) 5:3–11.[CrossRef][Web of Science][Medline]
Akamine A, Kida M. The effect of negative emotion on the reaction time: an event-related brain potentials study. Japanese Journal of Physiological Psychology and Psychophysiology (2001) 9:205–15.
Ambadar Z, Schooler JW, Cohn JF. Deciphering the enigmatic face: the importance of facial dynamics in interpreting subtle facial expressions. Psychological Science (2005) 16:403–10.[CrossRef][Web of Science][Medline]
Basso MR, Schefft BK, Ris MD, Dember WN. Mood and global-local visual processing. Journal of the International Neuropsychological Society (1996) 2:249–55.[Medline]
Bayliss AP, Frischen A, Fenske MJ, Tipper SP. Affective evaluations of objects are influenced by observed gaze direction and emotional expression. In: Cognition ((in press)) In Press.
Conway M, Giannopoulos C. Dysphoria and decision making: limited information use for evaluations of multiattribute targets. Journal of Personality and Social Psychology (1993) 64:613–23.[CrossRef][Web of Science][Medline]
Davidson RJ. Affective style and affective disorders: perspectives from affective neuroscience. Cognition & Emotion (1998) 12:307–30.[CrossRef][Web of Science]
Houwer J, Hermans D, Rothermund K, Wentura D. Affective priming of semantic categorisation responses. Cognition & Emotion (2002) 16:643–66.[CrossRef][Web of Science]
Derryberry D, Tucker DM. Motivating the focus of attention. In: The Heart's Eye: Emotional Influences in Perception and Attention—Niedenthal PM, Kitayama S, eds. (1994) San Diego, CA: Academic Press. 167–96.
Donchin E. Event-related brain potentials: a tool in the study of human information processing. In: Evoked Potentials and Behavior—Begleiter H, ed. (1979) New York: Plenum Press. 13–75.
Ekman P, Friesen WV. The Facial Action Coding System (1978) Palo Alto, CA: Consulting Psychologists Press.
Ekman P, Oster H. Facial expressions of emotion. Annual Review of Psychology (1979) 30:527–54.[CrossRef][Web of Science]
Fazio RH. On the automatic activation of associated evaluations: an overview. Cognition & Emotion (2001) 15:115–41.[Web of Science]
Fichtenholtz HM, Hopfinger JB, Graham R, Detwiler JM, LaBar KS. Event-related potentials reveal temporal staging of dynamic facial expression and gaze shift effects on attentional orienting. (2007) Presented at the annual meeting of the Cognitive Neuroscience Society: New York, NY.
Fredrikson M. Reliability and validity of some specific fear questionnaires. Scandinavian Journal of Psychology (1983) 24:331–4.[Web of Science][Medline]
Friesen CK, Kingstone A. The eyes have it! Reflexive orienting is triggered by nonpredictive gaze. Psychonomic Bulletin & Review (1998) 5:490–5.[Web of Science]
Graham R, Fichtenholtz HM, Friesen CK, LaBar K. Facial emotion facilitates target detection and identification but does not mediate reflexive orienting to gaze direction at short S.O.A.s. (2006) Presented at the annual meeting of the Cognitive Neuroscience Society: San Francisco, CA.
Hietanen JK, Leppanen JM. Does facial expression affect attention orienting by gaze direction cues? Journal of Experimental Psychology: Human Perception and Performance (2003) 29:1228–43.[CrossRef][Web of Science]
Hillman CH, Rosengren KS, Smith DP. Emotion and motivated behavior: postural adjustments to affective picture viewing. Biological Psychology (2004) 66:51–62.[CrossRef][Web of Science][Medline]
Holmes A, Vuilleumier P, Eimer M. The processing of emotional facial expression is gated by spatial attention: evidence from event-related brain potentials. Cognitive Brain Research (2003) 16:174–84.[CrossRef][Medline]
Hori E, Tazumi T, Umeno K, et al. Effects of facial expression on shared attention mechanisms. Physiology and Behavior (2005) 84:397–405.[CrossRef][Medline]
Isen AM. Positive affect. In: Handbook of Cognition and Emotion—Dalgleish T, Power MJ, Mick J, eds. (1999) New York, NY: John Wiley & Sons Ltd. 521–39.
Jasper HH. The ten-twenty electrode system of the International Federation. Electroencephalography and Clinical Neurophysiology (1958) 20:371–5.
Kienen G. Decision making under stress: scanning of alternatives under controllable and uncontrollable threats. Journal of Personality and Social Psychology (1987) 52:639–44.[CrossRef][Web of Science][Medline]
Klinnert MD, Campos JJ, Sorce JF, Emde RN, Svejda M. The development of social referencing in infancy. In: Emotion: Theory, Research, and Experience: Vol. 2. Emotion in Early Development—Plutchik R, Kellerman H, eds. (1983) 2. New York: Academic Press. 57–86.
Klorman R, Hastings JE, Weerts TC, Melamed BG, Lang PJ. Psychometric description of some specific-fear questionnaires. Behavior Therapy (1974) 5:401–9.[CrossRef][Web of Science]
LaBar KS, Crupain MJ, Voyvodic JB, McCarthy G. Dynamic perception of facial affect and identity in the human brain. Cerebral Cortex (2003) 13:1023–33.
Lang PJ, Bradley MM, Cuthbert BN. International Affective Picture System (IAPS): Instruction Manual and Affective Ratings (Tech. Rep. No. A-5) (2001) Gainesville: University of Florida, Center for Research in Psychophysiology.
Luck SJ. An Introduction to the Event-related Potential Technique (2005) Cambridge, MA: MIT Press.
Luck SJ, Woodman GF, Vogel EK. Event-related potential studies of attention. Trends in Cognitive Sciences (2000) 4:432–40.[CrossRef][Web of Science][Medline]
Mangun GR, Hillyard SA. Modulations of sensory-evoked brain potentials indicate changes in perceptual processing during visual-spatial priming. Journal of Experimental Psychology: Human Perception and Performance (1991) 17:1057–74.[CrossRef][Web of Science]
Mathews A, Fox E, Yeind J, Calder A. The face of fear: effects of eye gaze and emotion on visual attention. Visual Cognition (2003) 10:823–35.[CrossRef][Web of Science][Medline]
Maxwell JS, Shackman AJ, Davidson RJ. Unattended facial expressions asymmetrically bias the concurrent processing of non-emotional information. Journal of Cognitive Neuroscience (2005) 17:1386–95.[CrossRef][Web of Science][Medline]
Mogg K, Bradley BP. Some methodological issues in assessing attentional biases for threatening faces in anxiety: a replication study using a modified version of the probe detection task. Behaviour Research and Therapy (1999) 37:595–604.[CrossRef][Web of Science][Medline]
Mogg K, McNamara J, Powys M, Rawlinson H, Sieffer A, Bradley BP. Selective attention to threat: a test of two cognitive models. Cognition & Emotion (2000) 14:375–99.[CrossRef][Web of Science]
Musch J, Klauer KC. Locational uncertainty moderates affective congruency effects in the evaluative decision task. Cognition & Emotion (2001) 15:167–88.[Web of Science]
Putman P, Hermans E, van Honk J. Anxiety meets fear in perception of dynamic expressive gaze. Emotion (2006) 6:94–102.[CrossRef][Web of Science][Medline]
Schuller A.-M, Rossion B. Spatial attention triggered by eye gaze increases and speeds up early visual activity. Neuroreport (2001) 12:1–7.[CrossRef][Web of Science][Medline]
Schuller A.-M, Rossion B. Perception of static eye gaze direction facilitates subsequent early visual processing. Clinical Neurophysiology (2004) 115:1161–8.[CrossRef][Web of Science][Medline]
Schuller A.-M, Rossion B. Spatial attention triggered by eye gaze enhances and speeds up visual processing in upper and lower fields beyond early striate visual processing. Clinical Neurophysiology (2005) 116:2565–76.[CrossRef][Web of Science][Medline]
Schweinberger SR, Soukup GR. Asymmetric relationships among perceptions of facial identity, emotion, and facial speech. Journal of Experimental Psychology: Human Perception & Performance (1998) 24:1748–65.[CrossRef]
Sobotka SS, Davidson RJ, Senulis JA. Anterior brain electrical asymmetries in response to reward and punishment. Electroencephalography & Clinical Neurophysiology (1992) 83:236–47.[CrossRef][Web of Science][Medline]
Tipples J. Fear and fearfulness potentiate automatic orienting to eye gaze. Cognition and Emotion (2006) 20:309–20.[CrossRef]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||