Date event : December 13, 2018 - 2:00pm Conference / Talk Published on 11/19/2018 - 9:45am
The aim of this thesis is to describe face-voice multimodal integration during gender perception by using complementary behavioral and brain imaging techniques in light of recent principles of cortical connectivity (Markov et al., 2013b), such as the dual counterstream organization of feedback and feedforward connections and anatomically motivated hypotheses about large-scale cortical architecture such as generative models (e.g. Friston, 2005) and communication through coherence (Fries, 2015). Three structuring themes in particular were (1) how is multimodal integration affected by top-down and bottom-up effects? (2) Do task-specific modality dominance domains exist? and (3) Is it possible to identify the neural correlates of feedback and feedforward processes in multimodal integration?
Evaluating the influences of top-down attentional and bottom-up noise effects within a common signal-detection formulation requires similar weighting parameters for increasing attention or decreasing noise for one modality which increases the contribution of this modality and reduces the contribution of the other. This was empirically evaluated in two psychophysical studies using Maximum Likelihood Conjoint Measurement (Ho et al., 2008; Knoblauch and Maloney, 2012). In the first study 36 observers were directed to judge pairs of face-voice stimuli according to the gender of the face (face task), the voice (voice task) or the stimulus (stimulus task; no modality instruction given). While face and voice signals could never be completely disentangled (both modalities significantly contributed to all conditions), the top-down influence due to the task could be modeled as a differential weighting effect. In the second study, these results were reproduced and extended to 18 observers and the bottom-up effect of adding visual and/or auditory noise was also tested. Bottom-up noise could also be modeled as a weighting effect, as predicted by our theoretical results. This similarity of attention and noise effects might reflect shifts in functional hierarchy, defined (by analogy with anatomical hierarchy) as the proportion of feedforward versus feedback signal exchanged at a given time. The same change in this proportion could then be achieved in a bottom-up or top-down way by manipulating noise or attention, respectively. Communication through coherence offers a potential explanation for the neural basis of such a mechanism in terms of segregated supragranular and infragranular cortical streams which can be selectively modulated by oscillatory rhythm synchronization.
An asymmetry in favor of the auditory modality in terms of effect size was found in both behavioral experiments. Additionally, two independent (in the sense that they were significant in different conditions) interaction effects were found, first a multiplicative gender effect significant in the face and stimulus tasks, second an effect of gender coherence between modalities significant in the face and voice tasks. In a third experiment Dynamic Causal Modeling was used to analyze fMRI data (Friston, 2003). We investigated the modulation of effective connectivity between the Fusiform Face Area (FFA) and the Temporal Voice Area (TVA) in 12 participants while they were presented with multimodal face-voice stimuli and instructed to attend to either face, voice or any gender information. We found a change in effective connectivity in response to gender in the stimulus and face tasks (suggesting a link with behavioral gender interaction and task weighting effects). We also found a change in effective connectivity from the FFA to the TVA in response to gender incoherence for face and voice tasks (suggesting a link with behavioral coherence interaction). In terms of generative models, this would indicate the FFA as hierarchically lower than the TVA. The existence of two independent modulations points towards two independent hierarchical systems which could be supported by the anatomical dual counterstream architecture.
Derrington, Edmund Professeur UCBL Lyon1
van Wassenhove, Virginie DR CEA DRF/Joliot NeuroSpin
Mamassian, Pascal DR CNRS UMR 8248/LSP
Maertens, Marianne Technische Universtität Berlin
Papeo, Liuba CR CNRS UMR 5304/ISC
Dojat, Michel DR INSERM UMR 1216/GIN
Knoblauch, Kenneth DR INSERM UMR 1208/SBRI
Kennedy, Henry DR INSERM UMR 1208/SBRI