Adapting multimodal foundation models for recommendation

Fu, Junchen (2026) Adapting multimodal foundation models for recommendation. PhD thesis, University of Glasgow.

Abstract

Multimodal foundation models (MFMs) now show strong representation and generation capabilities. Adapting them to recommender systems is a natural and increasingly central research direction, since items often contain rich multimodal signals such as text, images, videos, and audio. Yet MFMs are designed for general representation and generation tasks, making specific recommendation-oriented adaptation a key challenge. This thesis addresses this challenge from two connected perspectives: representation adaptation and generation adaptation.

The first perspective studies end-to-end adaptation of continuous and discrete representations. Continuous representation adaptation jointly trains the MFM representation-learning pipeline and recommender through real-valued representations, making efficiency a central challenge. IISAN uses cached decoupled parameter-efficient fine-tuning to reduce computation overhead. Building on IISAN, we investigate adaptation depth and breadth. IISAN-Versa extends it to deeper, larger, asymmetric multimodal encoders, particularly pairing a large LLM-based text encoder with a smaller vision encoder. CROSSAN broadens it to multiple modality-specific foundation models through cross modal gated side adaptation. We then consider unified single-tower embedding models, whose general-domain capabilities pose a distinct challenge: existing multi-tower methods often become shallow side adapters and struggle with deep-layer adaptation. Stresa addresses this limitation through stream-aware residual adaptation. For discrete representation adaptation, jointly learning semantic-ID construction and the generative recommender introduces challenges, particularly non-differentiability and codebook collapse. DIGER tackles these using differentiable semantic IDs and an exploration mechanism that promotes codebook utilisation while maintaining end-to-end adaptation performance.

The second perspective addresses a gap in the literature: recommendation oriented generation adaptation remains underexplored from platform and user perspectives. On the platform side, although generation can complete missing modalities for item presentation and recommendation model training, its effectiveness in recommendation settings lacks evaluation. MMPCBench addresses this gap by benchmarking MFMs on multimodal product content completion. On the user side, generated content can enter the recommendation ecosystem, yet little is known about optimising it for engagement. LLMPopcorn investigates the popularity of MFMgenerated content and explores LLMs for popularity oriented micro-video creation. Across seven works, this thesis studies MFM adaptation across representation and generation.

• Item Type: Thesis (PhD)
• Qualification Level: Doctoral
• Subjects: T Technology > TK Electrical engineering. Electronics Nuclear engineering
• Colleges/Schools: College of Science and Engineering > School of Computing Science
• Supervisor's Name: Jose, Professor Joemon
• Date of Award: 2026
• Depositing User: Theses Team
• Unique ID: glathesis:2026-86308
• Copyright: Copyright of this thesis is held by the author.
• Date Deposited: 06 Nov 2026 15:57
• Last Modified: 06 Nov 2026 15:59
• Thesis DOI: 10.5525/gla.thesis.86308
• URI: https://theses.gla.ac.uk/id/eprint/86308

Related URLs

• Conference proceeding
• Article DOI
• Pre-print
• Pre-print
• Conference proceeding
• Article DOI
• Conference proceeding