Multi-task learning for effective Open-Retrieval Conversational Question Answering

Kongyoung, Sarawoot (2024) Multi-task learning for effective Open-Retrieval Conversational Question Answering. PhD thesis, University of Glasgow.

Abstract

Conversational Question Answering (ConvQA) is a rapidly growing area of research that aims to improve the search experience for users by allowing for more natural interactions between users and search systems. ConvQA systems are designed to gauge and answer questions in the context of a conversation, taking into account the previous questions and answers in the dialogue. One of the challenges of ConvQA is resolving ambiguities in the user’s questions based on the conversation history. This requires the system to not only consider the question being asked but to also take into account the conversation context to provide relevant and accurate answers. Open-Retrieval Conversational Question Answering (ORConvQA) is a more challenging variant of ConvQA, as it requires the system to retrieve relevant passages from a large collection of documents before extracting the required answers. This task requires the system to effectively search and retrieve the most relevant information, adding further complexity. In order to build an ORConvQA system, to address the ambiguities in conversational questions, a number of approaches have been proposed, such as follow-up question identification, conversational question rewriting, and asking clarifying questions. These approaches can help the system better gauge the user’s intent and context, thereby allowing it to generate more precise and relevant responses. Another challenge in ORConvQA is retrieving relevant passages from a large collection of documents and identifying the most relevant ones based on the conversation context. This is important because the extracted answers need to be based on the relevant passages, in order to ensure accuracy. On the other hand, Multi-Task Learning (MTL) has emerged as a promising approach to facilitate the learning of multiple related tasks by sharing the learner structure in a single model. MTL has gained considerable attention in recent years due to its effectiveness in addressing a diverse range of complex problems within a unified model. Therefore, we argue that learning ORConvQA approaches simultaneously can help to improve the system’s performance. In this thesis, we propose a novel ORConvQA framework leveraging Multi-Task Learning (MTL) to improve the performance of multiple related tasks by sharing their learned structure. By applying MTL to ORConvQA, we aim to leverage the benefits of addressing several related tasks to build a more effective and efficient model that addresses two main challenges: (i) ambiguities in conversational questions; and (ii) retrieving relevant passages from a large collection of documents before extracting the answers. To address ORConvQA effectively, we first propose an ORConvQA framework, which leverages a novel hybrid dynamic MTL method combining Abridged Linear for the main answer extraction task with a Loss-Balanced Task Weighting (LBTW) for the auxiliary related tasks, such as follow-up question identification, yes/no prediction, and unanswerable prediction, so as to automatically fine-tune task weighting during learning, ensuring that each of the tasks’ weights is adjusted by the relative importance of the different tasks. We conduct experiments using QuAC, a large-scale ConvQA dataset. Our results demonstrate the effectiveness of our proposed method, which significantly outperforms both the single-task learning and existing static task weighting methods with improvements ranging from +2.72% to +3.20% in F1 scores. Our findings also show that the performance of using MTL in developing the ORConvQA model is sensitive to the correct selection of the auxiliary tasks as well as to an adequate balancing of the loss rates of these tasks during training by using LBTW. To address the ambiguities in conversational questions, we propose the use of a text generation model with Multi-Task Learning for follow-up question identification and conversational question rewriting. Our derived models are based on text generation models –BART and T5–, and are trained to rewrite the conversational question and identify follow-up questions simultaneously. We evaluate our method using three test sets from the recent LIF (Learning to Identify Follow-up questions) dataset and a test set from the OR-QuAC dataset. Our results show that our proposed method significantly outperforms the single-task learning baselines on the LIF dataset, with statistically significant improvements ranging from +3.5% to +10.5% across all test sets, and also significantly outperforms the single-task learning of question rewriting models for passage retrieval on the OR-QuAC test set. Next, we employ an approach for asking clarifying questions to further address the ambiguities in conversational questions by proposing a novel hybrid method combining the generation and selection processes. Our method leverages Multi-Task Learning, combining the tasks of clarification need classification and the generation of the clarifying question to simultaneously determine when the initial user’s query necessitates a clarifying question and to generate a set of clarifying questions based on the user’s initial query and conversation history. A selection model is used to select the relevant questions from a question pool. To rank the candidate clarifying questions obtained from both the selection and generation approaches, the questions are scored using a text generation model for question classification. By using both the generation and selection approaches, our proposed method is able to generate a comprehensive set of questions while still ensuring that the selected question is relevant to the user’s queries. Our results on the TREC CAsT 2022 datasets demonstrate the effectiveness of our proposed method, which significantly outperforms existing strong baselines with improvements at P@1 by up to 20% on the relevance criteria and 30% on the novelty criteria. Finally, to effectively address our second challenge of retrieving relevant passages from a large collection of documents and extracting the answers, we propose monoQA, which uses a text generation model with Multi-Task Learning for both the reranker and reader. Our model, which is based on the T5 text generation model, is fine-tuned simultaneously for both reranking (in order to improve the precision of the top retrieved passages) and extracting the answer. Our results on the OR-QuAC and OR-CoQA datasets demonstrate the effectiveness of our proposed model, which significantly outperforms existing strong baselines with improvements ranging from +12.31% to +19.51% in MAP and from +5.70% to +23.34% in F1 on all used test sets. Overall, this thesis contributes an effective ORConvQA framework leveraging Multi-Task Learning to address the challenges of resolving ambiguities in conversational questions and retrieving relevant passages from a large collection of documents. Our proposed framework significantly outperforms existing strong baselines on a variety of benchmark datasets, demonstrating the effectiveness of MTL in improving the performance of ORConvQA models.

• Item Type: Thesis (PhD)
• Qualification Level: Doctoral
• Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science; Q Science > QA Mathematics > QA76 Computer software
• Colleges/Schools: College of Science and Engineering > School of Computing Science
• Supervisor's Name: Ounis, Professor Iadh and Macdonald, Professor Craig
• Date of Award: 2024
• Depositing User: Theses Team
• Unique ID: glathesis:2024-84305
• Copyright: Copyright of this thesis is held by the author.
• Date Deposited: 08 May 2024 14:19
• Last Modified: 08 May 2024 15:47
• Thesis DOI: 10.5525/gla.thesis.84305
• URI: https://theses.gla.ac.uk/id/eprint/84305