Forward and Inverse Rendering with Gradient-based Optimization

Other Titles



Photorealistic rendering is a fundamental component of computer graphics. To render photorealistic 2D images that faithfully represent our 3D world, efforts have been devoted into developing compact scientific models and efficient algorithms that are capable of simulating lighting and physics to capture appearance using the geometry, reflectance and illumination. Inverse Rendering, on the other hand, focuses on recovering unknown scene attributes from photograph measurements by optimization, typically through analysis-by-synthesis technique, or more recently, differentiable rendering. In this thesis, we propose solutions to a few such challenging forward and inverse graphics problems, as described below. The first work is a forward rendering (SIGGRAPH 2020) approach to speeding up Markov chain Monte Carlo (MCMC) rendering with derivatives from recent rendering engines that support end-to-end differentiation. In this work, we build upon Langevin dynamics and propose a suite of Langevin Monte Carlo algorithms with gradient-based adaptation for efficient photorealistic rendering of scenes with complex light transport effects, such as caustics, interreflections, and occlusions. In inverse rendering, we develop multiple solutions for recovering material and shape. In our second work (ICCP 2020) we focus on learning-based inverse subsurface scattering. Given images of transcluent objects, of unknown shape and lighting, we aim to use learning to infer the optical parameters controlling subsurface scattering of light inside the objects. With physics-based priors in the learning framework, we obtain strong improvements in both parameter estimation accuracy and appearance reproduction compared to traditional networks. Our third work is a unified shape and appearance reconstruction framework (EGSR 2021) using differentiable rendering for 3D object scanning. We tackle this problem by introducing a new analysis-by-synthesis technique capable of producing high-quality reconstructions through robust coarse-to-fine optimization and physics-based differentiable rendering. We demonstrate the effectiveness of our method on real-world objects captured with handheld cameras that outperforms previous state-of-the-art approaches. In Appendix A, we tackle the challenge of automatically generating procedural representation of fiber-based yarn models for cloth rendering (SIGGRAPH 2016, EGSR 2017). Appendix B has slightly deviated from rendering and focuses on image style transfer techniques (CVPR 2017, EGSR 2018). With recent advances in physics-based differentiable rendering, we have taken first steps to speeding up MCMC forward rendering with first-order gradients, improving learning-based inverse subsurface scattering, and introducing an end-to-end differentiable rendering pipeline for high-quality handheld object scanning. As for next steps, a promising future direction for computer graphics would be learning-based neural rendering methods that aim for AR/VR-based applications. Exploring differentiable rendering for improving traditional 3D reconstruction would be another promising topic, thanks to the theoretical and practical breakthroughs in the corresponding field.

Journal / Series

Volume & Issue


199 pages


Date Issued





Effective Date

Expiration Date




Union Local


Number of Workers

Committee Chair

Bala, Kavita

Committee Co-Chair

Committee Member

Bindel, David S.
Weinberger, Kilian Quirin
Marschner, Stephen Robert

Degree Discipline

Computer Science

Degree Name

Ph. D., Computer Science

Degree Level

Doctor of Philosophy

Related Version

Related DOI

Related To

Related Part

Based on Related Item

Has Other Format(s)

Part of Related Item

Related To

Related Publication(s)

Link(s) to Related Publication(s)


Link(s) to Reference(s)

Previously Published As

Government Document




Other Identifiers


Rights URI


dissertation or thesis

Accessibility Feature

Accessibility Hazard

Accessibility Summary

Link(s) to Catalog Record