Brains in dialogue: investigating accommodation in live conversational speech for both speech and EEG data.

Solanki, Vijay James (2017) Brains in dialogue: investigating accommodation in live conversational speech for both speech and EEG data. PhD thesis, University of Glasgow.

Abstract

One of the phenomena to emerge from the study of human spoken interaction is accommodation or the tendency of an individual’s speech patterning to shift relative to their interlocutor. Whilst the experimental approach to the detection of accommodation has a solid background in the literature, it tends to treat the process of accommodation as a black box. The general approach for the detection of accommodation in speech has been to record the speech of a given speaker prior to interaction and then again after an interaction. These two measures are then compared to the speech of the interlocutor to test for similarity. If the speech sample following interaction is more similar then we can say that accommodation has taken place. Part of the goal of this thesis is to evaluate whether it is possible to look into the black box of speech accommodation and measure it ‘in situ’.
Given that speech accommodation appears to take place as a result of interaction, it would be reasonable to assume that a similar effect might be observable in other areas contributing to a communicative interaction. The notion of an interacting dyad developing an increased degree of alignment over the course of an interaction has been proposed by psychologists. Theories have posited that alignment occurs at multiple levels of engagement, from broad levels of syntactic alignment down to phonetic levels of alignment. The use of speech accommodation as an anchor with which to track the evolution of change in the brain signal may prove to be one approach to investigating the claims made by these theories. The second part of this thesis aims to evaluate whether the phenomenon of accommodation is also observable in the form of electrical signals generated by the brain, measured using Electroencephalography (EEG). However, evaluating the change in the EEG signal over a continuous stretch of time is a hurdle that will need to be tackled. Traditionally, EEG methodologies involve averaging the signal over many repetitions of the same task. This is not a viable option when investigating communicative interaction.
Clearly the evaluation of accommodation in both speech and brain activity, especially for continuously unfolding phenomena such as accommodation, is a non-trivial task. In order to tackle this, an approach from speech recognition and computer science has been employed. The implementation of Hidden Markov Models (HMM) has been used to develop speech recognition systems and has also been used to detect fraudulent attempts to imitate the voice of others. Given that HMMs have successfully been employed to detect the imitation of another person’s speech they are a good candidate for being able to detect the movement towards or away from an interlocutor during the course of an interaction. In addition, the use of HMMs is non-domain specific, they can be used to evaluate any time-variant signal. This adaptability of the approach allows for it to also be applied to EEG signals in conjunction with the speech signal.
Two experiments are presented here. The behavioural experiment aims to evaluate the ability of a HMM based approach to detect accommodation by engaging pairs of female, Glaswegian speakers in the collaborative DiapixUK task. The results of their interactions are then evaluated from both a traditional phonetic standpoint, by assessing changes in Voice Onset Time (VOT) of stop consonants, formant values of vowels and speech rate over the course of an interaction and using the HMM based approach. The neural experiment looks to evaluate the ability of a HMM based approach to detect accommodation in both the speech signal and in brain activity.
The same experiment that was performed in Experiment 1 was repeated, with the addition of EEG caps to both participants. The data was then evaluated using the HMM based approach.
This thesis presents findings that suggest a function for speech accommodation that has not been explored in the past. This is done through the use of a novel, HMM based, holistic acoustic-phonetic measurement tool which produced consistent measures across both experiments. Further to this, the measurement tool is shown to have possible extended uses for EEG data. The use of the presented HMM based, holistic-acoustic measurement tool presents a novel contribution to the field for the measurement and evaluation of accommodation.

• Item Type: Thesis (PhD)
• Qualification Level: Doctoral
• Keywords: Speech accommodation, HMMs, EEG, phonetics, computing science, psychology, social signal processing, machine learning, dialogue, entrainment, neural entrainment, hidden Markov models, electroencephalography.
• Subjects: B Philosophy. Psychology. Religion > BF Psychology; P Language and Literature > P Philology. Linguistics; Q Science > QA Mathematics > QA75 Electronic computers. Computer science
• Colleges/Schools: College of Arts & Humanities > School of Critical Studies > English Language and Linguistics
• Supervisor's Name: Stuart-Smith, Professor Jane, Rachel, Dr. Smith and Alessandro, Dr. Vinciarelli
• Date of Award: 2017
• Depositing User: Mr Vijay Solanki
• Unique ID: glathesis:2017-8252
• Copyright: Copyright of this thesis is held by the author.
• Date Deposited: 16 Jun 2017 13:21
• Last Modified: 06 Apr 2018 15:33
• URI: https://theses.gla.ac.uk/id/eprint/8252