Neural correlates of multisensory decision making in humans

De Sousa, Gabriela (2021) Neural correlates of multisensory decision making in humans. PhD thesis, University of Glasgow.

Abstract

In everyday life we are constantly required to make decisions about things that we perceive in order to interact with our environment and perform everyday tasks. Perceptual decision making is the process by which sensory evidence is collected and accumulated towards one of two or more possible choices. This is an inherently a noisy process, and decisions made need to optimally trade-off between speed and accuracy, as well as combine complementary evidence of more than one sensory type (here, sound and vision). While great progress has been made in understanding the neural correlates of unisensory perceptual decision making, relatively little is known about the enhancements or changes to this process that result from the integration of more than one modality of information.

The current thesis presents empirical findings from three studies that sought to provide a more complete characterization of multisensory decision making using electrophysiological and diffusion modelling methods. Specifically, Study 1 (Chapter 2) investigates the temporal evolution of audiovisual decision making and compares whether early sensory integration or late post-sensory decision processing of visual evidence is enhanced in the presence of complementary auditory information. We recorded EEG measurements from human subjects during performance of a face versus car categorisation task. On some trials, participants were presented with images alone, while in others we simultaneously presented sounds of the same object category (i.e. speech and car sounds). Responses were more accurate and slower during audiovisual trials, and both accuracy and response time scaled with sensory evidence. Neural activity discriminating between face and car trials was observed peaking shortly before the time of response in a fashion that mirrored the process of evidence accumulation. This interpretation was confirmed using a neurally-informed drift diffusion model. Further, we found that trial-by-trial changes in behaviour could be predicted by neural activity within this model. Topographical representations of these signals revealed a prominent centroparietal cluster of activity.

Leading on from this, Study 2 (Chapter 3) modified a continuous version of the dot motion discrimination task to include sound motion and audiovisual motion trials. Participants received no obvious sign as to the start of a coherent motion period, which therefore prevented visually-evoked potentials and provided an unimpeded observation of evidence accumulation activity from the beginning of the trial to the point of decision. In doing so, we sought to further understand the enhancement of evidence accumulation activity during audiovisual trials. We focused on the same centroparietal cluster that we had observed in the previous chapter, and that was highlighted in the original study by Kelly and O’Connell (2013). Participants missed significantly fewer trials with audiovisual motion. Activity clearly increased at a steady rate from around 200/300ms post-stimulus onset, up until the point of response, in a pattern again mirroring evidence accumulation. We found that this activity was again enhanced during audiovisual trials compared to visual-only trials, with greater rates of increase in activity. Activity also peaked at a slightly higher level shortly before the time of response. These findings supported those of the Chapter 2 in that the presence of complementary auditory information enhanced the decision making process.

Finally, we asked whether oscillatory patterns within the EEG signals may offer additional insights into the neural representations of multisensory decision making. We extended the investigation of neural signals collected in Chapter 3 using the continuous dot motion discrimination task by decomposing the original broadband signal into its component frequencies, here focusing on beta, gamma, and high-gamma activity. We compared the rate of change in power between sensory conditions leading up to the time of response, as well as shortly after. While we did find interesting modulations in power relating to specific sensory conditions within the task, including a pattern of desynchronization that may suggest input from premotor structures in the embodiment of the decision, we did not find the same robust modulation in evidence accumulation by sensory condition that we had observed in the previous chapters. However, we could clearly see gradual changes in power that seems to reflect evidence accumulation.

Together, our results reveal novel insights into the neural representations of multisensory decision making in the human brain and point to new research directions that may uncover more about the neural underpinnings of audiovisual decision making. It also suggests further study of related activity such as decision confidence, or the embodiment of evidence accumulation within premotor areas.

• Item Type: Thesis (PhD)
• Qualification Level: Doctoral
• Subjects: R Medicine > RC Internal medicine > RC0321 Neuroscience. Biological psychiatry. Neuropsychiatry
• Colleges/Schools: College of Medical Veterinary and Life Sciences > School of Psychology & Neuroscience
• Funder's Name: Engineering and Physical Sciences Research Council (EPSRC)
• Supervisor's Name: Philiastides, Professor Marios
• Date of Award: 2021
• Depositing User: Theses Team
• Unique ID: glathesis:2021-82474
• Copyright: Copyright of this thesis is held by the author.
• Date Deposited: 05 Oct 2021 08:35
• Last Modified: 05 Oct 2021 08:47
• Thesis DOI: 10.5525/gla.thesis.82474
• URI: https://theses.gla.ac.uk/id/eprint/82474

Related URLs

• Article DOI