Concept-based interpretable framework for video-based affective signal processing

Li, Xinyu (2026) Concept-based interpretable framework for video-based affective signal processing. PhD thesis, University of Glasgow.

Abstract

Affective signal processing focuses on the computational analysis of human affect, behavior, and mental states through observable signals such as facial expressions, body movement, voice, and interaction dynamics. Over the past decade, advances in artificial intelligence, particularly deep learning, have substantially improved predictive performance across applications including human–computer interaction, healthcare, education, mental health assessment, and child development. However, affective AI is often used in sensitive decision-support settings, where model outputs may inform judgments made by clinicians, psychologists, educators, or social workers. In such contexts, predictive performance alone is insufficient. Models should also expose evidence that can be inspected, related to domain knowledge, and used to support responsible human oversight.

Most modern deep learning approaches to affective signal processing remain blackbox systems whose internal decision processes are difficult to interpret, validate, or relate to established affective theory. Existing post-hoc explainable AI methods can highlight influential pixels, regions, or latent features, but they often fail to identify what affective cues are being used or how those cues contribute to the final prediction. Feature-based approaches, by contrast, use semantically meaningful behavioral descriptors, but these are often fixed outside the learning process and may limit predictive performance. This thesis addresses this gap by asking how affective AI models can be designed to structure their predictions through intermediate concepts that are semantically meaningful, domain-grounded, and technically inspectable.

To answer this question, the thesis develops and evaluates a unified family of antehoc concept-based machine learning frameworks for affective signal processing. Concepts are treated as intermediate modeling components that mediate between raw sensory input and final task predictions. They are not assumed to provide human-centered interpretability by themselves; rather, they are evaluated through technical criteria such as task performance, concept alignment, spatial localization, robustness, concept removal, intervention behavior, temporal trajectories, and qualitative inspection.

The thesis first introduces a concept-based framework for Facial Expression Recognition, using facial Action Units as intermediate affective concepts. This framework shows that emotion prediction can be mediated through domain-defined facial behavior cues while maintaining competitive predictive performance. It then proposes the Attention-Guided Concept Model, which extends concept-based learning with spatial concept localization and multimodal concept fusion. This enables visual and acoustic concepts to be aligned across modalities for facial expression recognition and engagement estimation. Next, the thesis develops ConceptMamba, a concepttemporal model for video-based mental health assessment. By combining factorized concept tokenization with state-space sequence modeling, ConceptMamba captures long-range temporal dependencies while exposing how behavioral concepts evolve over time. Finally, the thesis extends concept-based modeling to interactioncentric affective computing through a graph-based framework for automated child– caregiver attachment disorder assessment. This framework uses de-identified multimodal interaction data and relational concepts such as interpersonal distance, body
direction, body openness, lean angle, and self-occlusion.

Across unimodal, multimodal, temporal, and relational settings, the findings show that concept-based models can remain competitive with strong black-box baselines while exposing domain-grounded concept-level evidence. The thesis therefore positions concept based modeling as a scalable design strategy for affective AI: not as a complete solution to human-centered interpretability, but as a technical foundation for systems whose predictions can be inspected through meaningful behavioral concepts. Future work should build on this foundation through stakeholder-centered evaluation, improved concept definition, fairness analysis, and causal validation.

• Item Type: Thesis (PhD)
• Qualification Level: Doctoral
• Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
• Colleges/Schools: College of Science and Engineering > School of Computing Science
• Supervisor's Name: Mahmoud, Dr. Marwa and Guha, Dr. Tanaya
• Date of Award: 2026
• Depositing User: Theses Team
• Unique ID: glathesis:2026-86023
• Copyright: Copyright of this thesis is held by the author.
• Date Deposited: 17 Jun 2026 11:26
• Last Modified: 17 Jun 2026 12:34
• Thesis DOI: 10.5525/gla.thesis.86023
• URI: https://theses.gla.ac.uk/id/eprint/86023

Related URLs

• Article DOI