Generative AI for 3D Content

Speaker:  Jun Gao, University of Michigan.

Time:        11:00 am, Jan.2

Location: SIST 1A-200

Host:        Jiayuan Gu

Abstract:

While generative AI models have achieved remarkable success in creating language, images, and videos, their applications in 3D content, which is the heart of several domains, such as VR/AR, film, gaming, and metaverse, encounter fundamental challenges due to the scarcity of 3D training data and increased complexities inherent in 3D. In this talk, I will present my research on developing 3D generative AI models to create realistic, high-quality, and diverse 3D content by leveraging the domain knowledge from computer graphics. First, I will discuss how incorporating the 3D modeling techniques from computer graphics could not only enhance efficiency and unlock new capabilities, but also regularize the generation behavior to focus specifically on the 3D geometry. Second, I will show how leveraging 2D foundation models could facilitate high-quality and diverse 3D content generation for geometry, appearance, and semantics by combining our 3D modeling techniques with differentiable rendering. The techniques we build turbocharge applications range from 3D generation from a single image, 3D reconstruction from multi-view images, 3D generative modeling, and text to 3D generation.  Finally, I will discuss the future direction of creating realistic 3D virtual worlds to enable immersive interactions.

Bio:

Jun Gao is a research scienst at NVIDIA and is an incoming assistant professor at the University of Michigan. His research lies in the interaction of computer vision, computer graphics, and machine learning, focusing on developing generative AI models to create 3D content for reconstructing, generating, and simulating lifelike 3D worlds.