An fMRI and psychophysical investigation of the temporal factor of audiovisual integration

Love, Scott (2011) An fMRI and psychophysical investigation of the temporal factor of audiovisual integration. PhD thesis, University of Glasgow.

Abstract

Our world is multisensory! To function and survive in our environment we utilise all of the cross-modal sensory information available to us. Some of it may be redundant but that simply makes our decisions more reliable (e.g., Clark and Yuille, 1990; Landy et al., 1995; Ernst and Banks, 2002). Knowledge of multisensory processes, which make use of this multitude of rich cross-modal information, is growing rapidly (for a review see, Calvert et al., 2004) and changing the way both philosophers (e.g., Macpherson, 2011) and scientists (e.g., Driver and Spence, 2000) think about perception and the senses.

One of the main challenges faced by this research field is how to decide which types of evidence are sufficient to prove that multisensory integration has occurred (e.g., Stein et al., 2009). To make things more complicated, the solution will be different dependent on whether integration is investigated at the neuronal, cerebral or behavioural level. Chapter 2 of this thesis provides new evidence about the integration criteria generally used in functional magnetic resonance imaging (fMRI). The choice of criteria had an obvious influence on which regions were found to integrate audiovisual information from the face and voice. Raising the question, which of these criteria should be used? Our conclusion was that it would be prudent to investigate the results of all of these criteria and; moreover, that a more cogent method of investigation would be to combine these criteria with manipulations of stimulus signal-to-noise ratio and/or congruency.

At the level of the neuron we know that the relative temporal synchrony of cross-modal cues can be an important factor in determining whether the neuron displays a multisensory response or not (e.g., Stein and Meredith, 1993). Manipulating temporal synchrony is actually one of the main techniques that researchers have used to explore multisensory processes at the neuronal, cerebral and behavioural levels (see relevant chapters in, Calvert et al., 2004). The experiments presented in Chapters 3, 4 and 5 of the current thesis focus on this temporal factor of audiovisual integration: the synchrony perception process in particular.

Chapters 3 and 4 present new evidence that two of the main experimental tasks used to investigate the synchrony perception process should not be used interchangeably. The experiments in those chapters asked participants to make both synchrony judgments (SJs) and temporal order judgments (TOJs) to identical audiovisual stimuli with various levels of cue asynchrony. First, Chapter 3 provided further evidence (Fujisaki and Nishida, 2009; van Eijk et al., 2008; Vatakis et al., 2008b; Vroomen and Stekelenburg, 2011) that these two tasks differ at the behavioural level. Second, Chapter 4 gave further credence to these behavioural results by using fMRI to localise, for the first time, the different neural correlates of these two tasks.

Moving beyond localisation of function, Chapter 5 outlines a fMRI experiment, which explored the role that different brain regions play in the synchrony perception process. Participants performed a SJ task to a large set of asynchronous audiovisual speech stimuli. Networks of regions responding preferentially to either synchronous or asynchronous audiovisual speech stimuli were found. However, which regions comprised these networks and whether they were both localised was dependent on which asynchronous conditions were included in contrasts of interest. This indicates that differences in stimulus and contrast choice in previous research could have helped produce inconsistencies in results (Stevenson et al., 2010).

• Item Type: Thesis (PhD)
• Qualification Level: Doctoral
• Additional Information: Due to copyright restrictions the full text of this thesis cannot be made available online. Access to the printed version is available once any embargo periods have expired.
• Keywords: multisensory, audiovisual, synchrony perception, fMRI, psychophysics
• Subjects: B Philosophy. Psychology. Religion > BF Psychology
• Colleges/Schools: College of Science and Engineering > School of Psychology
• Supervisor's Name: Pollick, Dr. Frank E.
• Date of Award: 2011
• Depositing User: Dr Scott Love
• Unique ID: glathesis:2011-3578
• Copyright: Copyright of this thesis is held by the author.
• Date Deposited: 28 Aug 2012
• Last Modified: 20 Oct 2017 12:26
• URI: https://theses.gla.ac.uk/id/eprint/3578