Shading concerned generative models for fine-grained photo-realistic image generation

He, Zhuo (2026) Shading concerned generative models for fine-grained photo-realistic image generation. PhD thesis, University of Glasgow.

Abstract

Modern generative models can synthesize highly realistic images, but their internal representation of content, material, illumination, and viewpoint is often implicit. This makes the generated result difficult to analyse and edit in a controlled manner. The central question of this thesis is therefore how to introduce fine-grained control into generative models without sacrificing visual fidelity.

This thesis studies that question through a sequence of methods that progressively connect controllable generation with physics-based rendering. First, we investigate controllability inside a style-based generator and introduce generative fields as a way to analyse the spatial extent of channel-wise control in StyleGAN2. This analysis is used to relate different generator layers to coarse and fine facial attributes, and motivates a reference-guided editing framework that preserves identity while transferring pose and expression from a reference image. The resulting study shows that controllability inside a latent generator can be made more explicit when the spatial role of intermediate channels is analysed rather than treated as a purely implicit property of the model.

Second, we study illumination control as a rendering problem and propose a physics-based neural deferred shading pipeline for real-world portrait images. The method takes estimated material, geometry, and illumination attributes as input, and learns a scene-agnostic mapping from geomertry buffer representations to photorealistic shading under HDR environment lighting. To support this study, we construct FFHQ256-PBR, a large facial dataset with estimated PBR textures, geometry buffers, lighting, and camera parameters. This part of the thesis shows that a learned shader can compensate for the mismatch between inverse-rendered real-world attributes and classical rendering assumptions, making relighting more controllable in practice.

Third, we integrate this rendering perspective into text-to-image generation. We introduce ShadingFusion, a rendering-aware diffusion pipeline in which a modified latent diffusion model predicts decomposed material representations rather than only final RGB appearance, and a neural shader then renders the result under controllable illumination. To enable this setting, we construct a paired dataset of portrait images, estimated PBR attributes, and structured text descriptions, and we redesign the VAE decoder as a multi-head architecture that jointly reconstructs RGB content and G-buffer outputs. This decomposition allows diffusion-based synthesis to retain photorealism while supporting explicit relighting and material-aware control after generation.

Finally, we extend the same idea to 3D generation. We propose DiffGSPBR, which combines generative 3D Gaussian splatting with deferred shading to produce decomposed 3D scenes whose materials, lighting, and viewpoint can be edited after generation. Built on top of a generative Gaussian-splatting backbone, the method adds a material-estimation head, a global illumination-estimation head, and a physics-based Gaussian deferred renderer that closes the loop through self-supervised reconstruction. In this way, the thesis moves from controllable 2D generation to editable 3D scene synthesis, showing that rendering-aware decomposition can support relightable and view-consistent generation in a unified framework.

Taken together, these contributions show that separating content generation from image formation is a practical way to improve the interpretability and controllability of generative models, rather than treating rendering as an implicit by-product of sampling. This thesis demonstrates that rendering can be brought back into the generative pipeline as an explicit and editable component for both 2D and 3D synthesis.

• Item Type: Thesis (PhD)
• Qualification Level: Doctoral
• Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
• Colleges/Schools: College of Science and Engineering > School of Computing Science
• Supervisor's Name: Pugeault, Dr. Nicolas and Henderson, Dr. Paul
• Date of Award: 2026
• Depositing User: Theses Team
• Unique ID: glathesis:2026-86093
• Copyright: Copyright of this thesis is held by the author.
• Date Deposited: 10 Jul 2026 14:08
• Last Modified: 10 Jul 2026 14:12
• URI: https://theses.gla.ac.uk/id/eprint/86093

Related URLs

• Conference proceeding
• Pre-print