Transcript

23718 3D 4D Generation and Modeling with Generative Priors

From ComputerVisionFoundation Videos · Episode page · Watch on YouTube

July 5, 2024 · 1 h 31 min · 11,760 words · 77 timestamped sections

This is the full, timestamped transcript of "23718 3D 4D Generation and Modeling with Generative Priors", an episode of ComputerVisionFoundation Videos published July 5, 2024 (1 h 31 min, about 11,760 words). It mentions 12 people, works and ideas, including 4D Generation, Score Distillation Sampling, 3D Gaussian Splatting, Nerf, Shingly.

Who speaks in this episode?

Shingly (Host)
P. Jong (Host)

What is discussed, and when

0:00 – 10:00: 4D Generation, Nerf, Shingly, Snap Research
20:00 – 30:00: P. Jong
30:00 – 40:00: Score Distillation Sampling
1:00:00 – 1:10:00: 4D Generation, Snap Research
1:10:00 – 1:20:00: Depth Estimation, 3D Gaussian Splatting, Nerf, Mosa
1:20:00 – 1:30:00: Score Distillation Sampling, 4D Generation
1:30:00 – 1:31:08: 4D Generation, Score Distillation Sampling

Notable quotes from "23718 3D 4D Generation and Modeling with Generative Priors"

For the 34 generation and modeling with genetic prior so I am shingly from the snap research and today's tutorial will be hosted by me p and Chan so let's get started to briefly explain what we can expect from today's.
about Shingly · at 0:03
Marching with this neuron Network here is the visualizations provided by the original Nerf paper so caring the point in the space and then through the re marchings we can get the image and we can use the image.
about Nerf · at 7:08
Get high quality 3D object generation in this tutorial we will Begin by leveraging large 2D sets using diffusion models for 3D generation then we will explore how to use 2D PRI…
about P. Jong · at 22:05
These images One straightforward way is to use the model for multi view score distillation sampling or SDS in this case we consider that the multiv view generator has learned the 3D prior from us from the 3D data and could.
about Score Distillation Sampling · at 36:02
Introductions and let's take a quick break and then let's be back to the final part this is cyang from snap research and in this session I will talk about 4D generation and…
about Snap Research · at 1:01:58
Let's first have an abstract view of what 4D generation is about at a very high level 4D generation can be oversimplified as a method capable of generating frames at different…
about 4D Generation · at 1:04:42
Point is a gaussian unlike Nerf which requires Ray tracing and point sampling gaussian splatting uses rasterization making it much cheaper to compute while still achieving…
about 3D Gaussian Splatting · at 1:11:35
Signal to use is depth map estimated from a monocular depth estimation network using depth maps estimated by a pre-trained depth estimation network is perhaps the most important component for robust 4D reconstruction this is because as we.
about Depth Estimation · at 1:13:43

Who and what is mentioned in this episode?

People

Shingly — Person · Main speaker / tutorial host (segment 1) · 1 mention · first at 0:06
Introduces and frames the tutorial agenda on 3D/4D generation and modeling, speaking as a Snap Research representative and setting up the tutorial structure and motivations.
P. Jong — Person · Main speaker / tutorial host (segment 2) · 1 mention · first at 20:49
Presents the portion of the tutorial focused on leveraging large-scale 2D diffusion priors for 3D generation, including SDS and multi-view-to-3D pipelines, and discusses limitations and feed-forward approaches.

Companies & orgs

Snap Research — Company · Affiliation · 2 mentions · first at 0:06
The research division of Snap Inc., involved in cutting-edge 3D and 4D generation technologies.

Products & tech

Bano — Product · Research project · 3 mentions
A method for reconstructing articulated objects like cats and dogs from videos.
By synthesis now let's review a few state-of-the-arts 4D reconstruction methods the first method is Bano which is able to reconstruct articulated objects like cats and dogs…
Mosa — Product · Research project · 1 mention · first at 1:17:28
A method for reconstructing generic dynamic scenes, achieving high-quality novel view synthesis.
Is supervised by Optical flows segmentation masks and also continuous surface embeddings a more recent method mosa has been developed for reconstructing generic Dynamic scenes…
Vision Pro — Product · Example hardware · 1 mention · first at 1:28
A hardware device mentioned as an example for viewing 3D and 4D content.
Interesting pioneering attempts are already been there for example like this kind of using a Vision Pro to help the creation process including like 3D Generations or like a 3D…
Tesla Model X — Product · Example object · 1 mention · first at 1:13:27
Mentioned as an example of an object that can be animated using 4D generation techniques.
Easily control and manipulate different parts of the gaussian independently this allows us to create interesting animations like this one where a Tesla Model X waves its doors…

Concepts & topics

4D Generation — Concept · Core topic · 11 mentions · first at 9:19
Central theme of the episode, discussing the creation and evolution of 4D content using generative AI.
Score Distillation Sampling — Concept · Key technique · 6 mentions · first at 36:07
A method for optimizing 3D representations using pre-trained 2D diffusion models, crucial for 4D generation.
3D Gaussian Splatting — Concept · Key technique · 3 mentions · first at 1:11:27
A modern technique for 3D representation, enabling faster and more efficient rendering compared to NeRF.
Nerf — Concept · Key technique · 2 mentions · first at 7:14
Neural Radiance Fields (NeRF) is a foundational technique for 3D and 4D generation, enabling photorealistic renderings.
Depth Estimation — Concept · Key technique · 5 mentions · first at 1:10:32
Essential for 3D and 4D reconstruction, providing depth information from monocular images or videos.

Full transcript with timestamps

0:00 hi everyone uh Welcome to our tutorial uh for the uh 34 generation and modeling with genetic prior so I am shingly from the snap research and uh today's tutorial will be hosted by me p and Chan so uh let's get started uh to briefly explain what we can expect from today's tutorial uh our mission is where I believe uh many uh most of us like doing the 3D 4D modeling uh is to digitile everything and generate anything so uh 40 modeling can be used in VR setup like these movies and gamings it can also be used in like AR setup uh viewing with Hardwares devices or for construction to provide more immersive uh experience so uh the medium that we are the most familiar with is the phone screen or uh through computer screens and I believe everyone has seen this kind of uh lenses or visual effects which usually require 3D knowledge um however obviously 3D is not just a better 2D and just to help create uh

1:11 better 2D so recently there are uh a lot of like headware devices and some um interesting pioneering attempts uh uh are already been there uh for example like this kind of uh using a Vision Pro to help the creation process including like 3D Generations or like a 3D painting and drawing or of course it can be uh really helpful in uh for the educational purpose uh like this kind of uh to see what inside the uh the machine or can help some skill uh practicing like a piano so what does 4D consist of um uh so we have like static 3D object uh we have a background 3D scene and we have Dynamic objects stat scenes so we will cover all of these today let's talk a bit more about the evolution of 3D uh start from uh when we only have limited 2D and 3D data the inference is fast but uh the quality and diversity are limited people then realize will never get enough 3D data and at a time with the emergence of a

2:33 large scale image diffusion model people found a way to distill Knowledge from these 2D models the quality is amazing but this optimization based methods are slow taking up to hours to generate a sample more recently a fif forward Paradigm with multiv view images as intermediate medium has become popular today we'll cover all this on the other hand uh for the generation is in its very early stage currently the speed are slow and quality is not as good as to DM 3D uh we are making some exciting progress recently uh we'll also cover that uh what's the next step of uh uh for generation is uh still unclear whether we will focus on like making it faster well we'll focus on like boosting the quality uh we we'll see finally I want to uh bring up an ongoing trend of interaction between reconstruction and generation conventionally reconstruction and generation are separated task reconstruction is like understanding the

3:42 world and generation is creating the world however recently with the help of amazing image and video models these two text are now highly collaborative uh first uh Generations via reconstructions we can perform 3D or 4D Generations by two stage methods first generate like multiv view image or video Then followed by the three for constructions on the other hand those uh reconstructed 3D and 4D uh samples can potentially be used as training data for the uh fif for the future fif forward 3D and for the unity model uh to overcome the data limitations okay so here's uh the uh agenda for today's tutorial uh because this is a tutorial so we will not uh going to dive into technical details of specific paper uh but focus on the overview of the uh of of the field so but feel free to raise any questions for online or offline uh and to discuss with us after the tutorial so this part uh we are going to talk

5:03 about 3D generation without large scale to depress I know uh this is not the topic that most of you uh today come for but uh I will still cover what uh conventionally people were doing before this large scale uh generation model era for the sake of completeness so uh like centuries ago before we have large scale image model and before we know how to interact among 2D 3D data and 2D 3D models the way of training 3D generating models can be easily divided to trending using 2D data and trending using 3D data let's start from uh using 2D data so uh 2D data is the special sampling of 3D to learn 3D with only 2D data we need a way of modeling uh this simpling process there are two important technique contributing to this directions first algorithm wise nerve what nerve provided is a uh continuous implicit scene representations and a neuron network based differentiable volume rendering and architecture wise we have

6:19 Stan before the diffus uh diffusion model era Stan was dominating a generative field with several important characteristic first the style base architecture provides the ability of disentanglement of attributes and disentanglement of different levels of details until now uh in certain domains like human phases diffusion model still cannot achieve the same level of control compared to stame Second it shows the importance of improved Laten space let's quickly go through these two uh neural neural Rance field model the 3D scene with an implicit representation that is a neuron Network that we can query the color and opacity with 3D location and Vie angles then we can perform differentiable rendering with Ray marching uh with this neuron Network here is the visualizations uh provided by uh the original Nerf paper so caring the point in the space and then through the re marchings we can get the image and we can use uh the

7:29 image uh to get reconstructions uh laws and then to optimize the 3D representations next stalen uh Stan was a revolutionary architecture that achieve product level Genera result on certain domain like human faces they propose a mapping Network to obtain better Laten space to sample from and a style based architecture that use ad in to inject style uh into different layers in the network so how did people make use of these two techniques to learn 3D from 2D before Nerf holog again did the pioneering uh attempts using Stan Inspire constant uh Laten and sty injection along with 3D transform and projection module after that almost most methods almost all methods adopt the radiance field and random technique like graph uh VPR 21 base paper giraffe pyam Vol Gam gram Etc uh sound of them make use of s of the stal and techniques in their pipeline finally a method successfully Mar release to instead of adopting Stan

8:44 technique to whatever pipeline sty nerve starts starts from sty gate architecture itself and incorporate Nerf architecture the visual results improves substantially which can also be observed in quantitative numbers later on the sem work e3d became the uh using the b word nowadays Foundation model for 3D aware generation test the most important techniques they propose uh is the tri plan representations which is still playing crucial role in current 3D even 4D generation neuro implicit representation use a fully connected layer to represent a scene uh which can be quite slow to query on the other hand explicit musle Grid or hybrid variants use small implicit decoders that are fast to query but uh it is it scale poorly with resolution and Trion is like a in the in the middle so it's a hyper representation that is fast and scale efficiently with resolution enabling uh to have better detail for uh equal

9:54 capacity the tri print representation involves uh using three orthogonal 2D feature PL to encode uh the 3D scene so the the whole each 3D architecture consists of a neural network Trend to map noise to uh three channels uh Nerf rendering uh to to uh to get a get a 2D image from uh triples and randoming Sample camera angles followed by the upsampler uh to uh to increase the resolutions so again uh these Tri representations and the uh and the usage are still quite common and popular in nowaday 3D generation pipeline here are here are some results uh you can see basically uh they inherit the style G quality but now make it uh 3D aware it also inherits the manipulation uh capability of sty G and of course numerically it outperforms uh previous methods with a great margins speaking of manipulations and Stan sty control ability of St 3D previously uh we had this interesting project trying to learn 3D of those

11:22 artistic domains uh where the ground true 3D does not even exist like cature and pixza and uh we can perform attrib editing in 3D just like what stalen can do okay uh let's get back to e3d what are the issues of e3d first camera so the camera uh parameters are sample from a manually chosen uh distribution and second the geometry the geometry is learned at a rather low uh resolution so as we all notice most results are from specific domains like faces and cats why so it's because uh we uh have a strong assumption on the data so the objects need to be alignable and Camera distribution in the data uh should be simple what does it mean uh so alignable means that each phase need to be roughly put in uh the center and it need to have easy to Define camera Post Distribution uh the original h3d paper uh spend uh substantial uh effort in like curating and uh cleaning the data to make sure uh the model is

12:44 trainable so how about uh if like the image net uh data so objects has different size and maybe they have they are like multiple objects in the image so how can we generate Le 3D aware result on this kind of data set uh we developed uh like two years ago we develop 3D GP uh it is the first 3D aware G model that can train on a thousand diverse uh category High Lely we add a camera control and a generated image prior to improve the quality uh so instead of like having a predefined uh uh camera distributions we now have a learnable uh camera uh camera prior camera generator and then to improve the geometry we uh using the depth condition in uh into the discriminator uh and we propose these depths adapter which the overshot generator they predict relative deps and the depths are not ideal so in order to align the two distributions uh we add a trainable adapter uh to to make this happen so here are the result we can

14:04 clearly see the difference from the 3dgp uh to the original e DD and here are some animated visualizations to make it clearer but uh you may argue that it is not a real 3D generation right it is just like with a limited camera control and uh it is still image based so uh often we call it like 3D aware image Generations or sometime we just jokingly call it like 2.5d Generations now this uh this lead us to the next branch which directly uh train 3D Generations on 3D data uh with 3D data it is definitely uh 3D generation right now let's start from 3D representations uh there are quite a lot of 3D stations each with their pros and cons the most commonly used are PL Cloud vxo measures and implicit neural uh of course there are uh lots of other representations like uh part level representations uh and they are uh cat uh computer edit design uh but we'll focus on uh these two this for today and

15:25 on the other hand uh 3D Generations using 3D data has closely follow the advancements of the 2D Genera model all the way from VA again to the recent diffusion and Transformer in the field of uh training 3D using 3D data the way of Cate uh categorizing methods is uh fairly simple basically uh in these two list we pick one representations and uh one backbone uh you will find corresponding uh works almost so high Lev speaking the motivations simple uh that is trying to replace the 3D data with uh uh replace the 2D data with the 3D uh representations by modifying architectures uh like making 2D CNN 3D CNN or like a point cloud based convolutions then uh the rest are very similar to the original 2D versions so I will not go into details in this part uh but I will go over some important uh literature first uh PL Cloud plus again uh this icml work was one of the first attempt in the more than uh uh 3D

16:39 Generations then with normalizing FL uh flow the quality improved with improve uh backbone architectures uh the normalizing flow is a series of invertible mappings that transform an initial non distributions to a more complicated one uh more recently of course the fusion model uh the quality further improve next uh with vxo uh this was a very early work even before uh the shipet data set that used the boxo plus again even with the old school vae with some smart design to take the characteristics into account we can get some interesting results that are unique uh to 3D it is noteworthy that the neuro field as an implicit representation is usually difficult to learn directly uh with uh conventional methods so usually we'll try to uh sample and veliz it uh for example uh with the Transformer this Auto SDF it first convert SDF into vxo grid and then treat each grid as a token and train a model using

18:01 Transformer and next of course uh with diffusion SD Fusion similarly voxelize the SDF then Trend an auto encoder to encode them into a latent space nowadays everything in a latent space with an auto encoder can uh be train uh using a latent diffusion model finally uh MH uh in this cvpr before mesh GPT will be presented it is a paper I really like as it works with mesh which is the most direct representation for all Downstream tests and it's tokenized and Trend in GPT manner which has greatly uh has great potential to work together with other modalities first it performs Vector quantizations to learn token and uh it just Trend uh this token uh using this token to trend of generation in the GPT uh manner here's a demo uh released by the authors to kind of like demonstrate how this kind of Auto regressive um like MH base Generations can be used in um the like object

19:26 Generations okay so uh you may notice is that until now most of the generation 3D generation results only shows like uh the simple Furnitures uh at most uh car and some vehicles uh it is be because that before recently people that's all we have so uh people had been using gemet all the time and until recently we have uh this object vers XL but uh we will uh compare to the large scale image data said that up to several billions uh at least given the current Hardware capturing Hardware device and capturing Technique we in the short term we will never get enough diverse and imaginary uh data uh compar in in 3D compar to Tod and also because like currently all the users and researcher are quite spoiled uh we do not want uh simple say a cad or simple uh chair we want say uh El dancing on a chair eating ice cream or something like that so which leads us to the next topic uh next topic we inevitably need to uh

20:40 make use of uh Le large scale uh to the uh image priors hello everyone my name is p Jong and I'm a research scientist at snap today I'm excited to present a tutorial on 3D generation this tutorial will cover the journey from per instance optimization to fit forward techniques 3D generation is a task that aims to automate the creation of 3D objects similar to what digital designers or our designers do the 3D objects can be generated by some methods that either take T text prompts as input or take a single image as input how do those methods work using 3D data to accomplish 3D generation tasks is a straightforward way in recent years the community has collected large scale 3D data sets from diverse object classes and with complex appearances and shapes however this 3D dat set are still significantly smaller than the available 2D dat set so utilizing large scale 2D data is nowadays A crucial strategy in order to

22:05 get high quality 3D object generation in this tutorial we will Begin by leveraging large 2D sets using diffusion models for 3D generation then we will explore how to use 2D PRI or combine 3D DSS with 2D DSS to enhance 3D generation in the more efficient way let's start by exploring 2D diffusion the main idea is to map a pre predefined noise distribution to an image distribution this process can be challenging instead a NE way is to decompose the mapping step by step and in each step a random noise is progressively added to an image and the image becomes pure noise sample this is known as diffusion or the forward process by traversing this trajectory backwards gradually adding the inverse noise sample we can get back the original image this is called the reverse process if we have a data set that contains multiple images we can learn a general noise predictor as marked in the red box the noise predictor takes a noisy

23:37 image and a time stamp as input and predicts the added noise additionally people can control the generation process by conditioning the model with a text prompt or an image 2D image dat set can be massive and inherently contain 3D information about objects for example a picture of a cat can be taken from multiple angles and show different body shapes given the reach 3D information in 2D image dats we are able to learn 3D generation using 2D diffusion models this optimization problem can be formulated in this way assume a 3D representation is denoted as Theta and the rendered images are denoted as X using some render techniques called G the goal is to use a pre-trained 2d diffusion model to update the 3D representation Theta so that the rended results X follow the given instructions whether the instruction is a text prompt or an image one optimization approach for this is called Square distillation simply or

25:09 SDS SDS involves rendering images from 3D representation Theta adding noise to the image then Den noising the image the residual between the input and the the denoise image is a score which is back propagated to the 3D representation Theta here on the left we visualized the SDS optimization process for an object on the right you can see more examples of the final optimized 3D results however one issue of this SDS optimization is that even a text prompt SDS almost can only generate a single mode of a 3D object in other words there's no diversity one further direction to improve SDS is to generate diverse results for one input text prompt as shown in this slide to achieve this instead of optimizing a single Point Theta people learn the distribution of theta denoted as Mu and generate diverse results by sampling Theta from this distribution people learn this distribution me using variational

26:46 inference basically we can get a set of rendered images from the Learned SATA distribution the rendered images represent the variational distribution the goal is to match this variational distribution to the Target distribution that consists of the gr TRS images here we assume that the pre-trained 2D diffusion model outputs the target distribution the question now is how to optimize this distribution matching directly matching two distributions is challenging because the images are at complex and high dimensional space in contrast optimizing on the noisy distribution is more feasible as the noise distributions are known and predefined so the optimization goal can be changed to matching the noisy distributions over the time steps making the training process more manage manageable and effective another important direction is to generate more photorealistic 3D like what I show in this Slide the given text

28:07 prompt is below each object recap that we get noisy distribution input by blending noise to a clean image or a clean latent Vector if we use lat diffusion models the noise schedulers r t and Sigma T are functions of time step in the original SDS work the temp step is randomly sampled over the training iterations however random sampling is not ideal but adding large noise at the latter training stages can result in diverge Den noising output and lead to the change or average of the 3D shapes and alternative way is temp step maning instead of random sampling the simple temp step is determined and decreased linearly over training iterations the results now look better than uh those obtained with random tab step sampling in some papers a square root sampling strategy is proposed the idea is that few fewer optimization steps are needed at the beginning but more steps are needed with smaller time steps in order to settle

29:36 down the geometry quickly and refine the texture details with more steps we mentioned that all the stretches are based on empirical evaluation there's no theoretical proof indicating that which temp step sampling strategy is optimal in SDS the above Works using nerves as 3D replantation in practice however this could lead to bumpy geometry when extracting mashes from nerves using marching cubes to avoid that people also directly use mchh representations for better geometry and then optimize the appearances and material properties of this meshes this leads to better visual effects and Mak it more adaptable to Downstream 3D tasks that directly need mashes moreover another issue about Nerf is that high resolution rendering from Nerfs is computationally expensive to address this magic 3D first generat low resolution Radiance field extract 3D mesh and continues to refine the 3D mesh using SDS the two-stage approach provides high

31:13 resolution 3D mesh models for faster training Speed 3D Gan splatting is a better representation because it enables quicker optimization and faster convergence although there are some achievements several limitations still exist when using 2D priers for 3D generation one significant issue is the yess problem or multi face problem where multiple faces are generated from different views moreover getting each 3D object within minutes is still relatively slow methods to generate objects in seconds or even in real time is more preferred with quickly review the literature on 3D generation using 2D priers like using score dissolution sampling and its variants also there are additional ways to achieve better 3D geometry now let's move from using 2D prior only to combining 3D as with 2D priers for faster generation one Paradigm is to decompose the task into two stages at the first stage diffusion models are either

32:57 between from scratch or fine T to generate multiv view images at the first at the second stage reconstruction techniques are used to reconstruct 3D from multiv view images generating multiple views using a model this can be achieved by a synchronized multiv view diffusion model which captures The Joint probability distribution of mulle view multiv view images The multiv View diffusion model is achieved by expanding a single image diffusion model into a multiv view version the model can be fed on 3D data set like object wordss in order to add more 3D information into the multiv vi generation process this work added tripl features as intermediate representation during The multiv View image diffusion process in this case a Transformer takes noisy multiv view images as input and predicts a clean trap plan feature as output and it leaves it adds slightly um less noise back on the render images for

34:18 the next step the noising iterations another representative method beuse a special feature volume for the multiv views which is then fed into the diffusion unit as attention features to generate final Target views here is a inference process for generating multiple views

35:00 in addition to generating RGB images we can also generate auxiliary geometric information this work generates surface normal Maps along with the RGB images we mentioned that beyond the surface normals there are also other geometric information that can also be used for training diffusion such as deps albo and 3D coordinate Maps instead of generating Prett Define Target views in a synchronized manner another approach generates infinite views by conditioning the diffusion models with Target cameras and a source image view by doing so it can generate multiple views in sequence so with multiv view images at a hand how to reconstruct 3D shapes from these images One straightforward way is to use the model for multi view score distillation sampling or SDS in this case we consider that the multiv view generator has learned the 3D prior from us from the 3D data and could produce useful multiv view SDS gr

36:23 gradients for 3D optimization on the left it shows the SDS results using 2D PRI only which suffers from the yess problem and generate M multiple phes the inconsistent and unrealistic view generation happens due to the lack of 3D structural understanding in the middle the improvements is brought by incorporating 3D priers the geometry is much better and the appearances are much more um accurate but remains somewhat Below on the right using both 2D and 3D prior provides overall best Geometry and appearance recent Works further improved the the appearance quality by combining the 2D and 3D PRI with improved SS moreover some personalization techniques such as dream boost is also used to maintain the Fidelity of the appearance reconstruction here are um some image to 3D reconstruction results using this

37:47 approach alternatively with auxil auxiliary geometric information such as surface normals we can directly reconstruct 3D objects using 3D reconstruction techniques like news the 3D rendering results of this approach are shown at bottom right so far the two multiv view to 3D reconstruction methods we've introduced are still relatively slow for faster reconstruction researchers have proposed to uh fit forward models that can reconstruct 3D shapes from multi view images in seconds one example approach represents multiv view images as a TRL feature and decoding this Tri feature into Radiance views via volume rendering this approach is joined on object verse data set thus it enables to do General 3D uh reconstruction in the fifth hour way the visualized 3D results are at bottom to get such a fit forward reconstruction pip planine the multi view images are given into a Transformer and being used

39:07 as keys and values learnable trap plant tokens are used to query the image features and produce a trap plant feature after having the Trap plant features volume rendering techniques are applied to convert this features into Radiance field similar to SDS we just mentioned that extracting meshes from Radiance FS often lead to a loss in a visual quality so some approaches generate Radiance views as an initial step and then fine-tune the entire model by replacing the radiance field representation with differentiable mhes to further enhance the mash appearance additional post fine tuning steps can be used specifically this work fine tunes the color MLP and the TR representation this fine tuning step could be efficient taking only about 4 seconds for one instance yet it is significantly improved appearance reconstruction details here are the spars view reconstruction results together with image or text to

40:23 3D generation results we also mentioned a few more concurrent works for the fit forward match generation task in this two Works one's use differentiable Marchin Cube or extracting meshes the other one use is flex Cube which is a hybrid implicit and explicit 3D MH representation we won't go uh into two details for this two Works beyond that gous spatting can also serve as an alternative representation in the fit forward work this work designs a fit forward model to predict 3D structures from multiv view images through 3D Gan splatting and here are the results

41:42 well this text or image to multiv view and then multiv view to 3D pthan can provide good results to some extent it still has failure cases here and there one significant issue is the inconsistent multiv view image generation in this case the generated novel views are not well aligned with with each other view leading to inconsistency issues in 3D reconstruction alternatively people borrow more priors from other data modality such as videos in this case a video diffusion model is f tuned on multiv images and output static object views for 3D reconstruction one months ago a new or came out that enables photo realistic multiv view generation the pipline can take any number of images and any specific novel view points as input and generate 3D consistent images as output to achieve this a masked multiv view diffusion models is proposed in this work moreover instead of only using object words for f tuning this word

42:55 combines four real world multiv view as as training data here are the visualizations of this um uh this work alternatively a very recent work came out a few days ago proposes single stage three generation in other words there's no not multiv images as intermediate representation during inference the model takes an image or a text prompt has input and directly generates meshes this model architecture is similar to 2D diffusion models but lifts up to 3D space specifically a Transformer encoder and a convolutional decoder consists the vae encoder backbone where the input is point clause and the output is Tri plan the vae efficiently encodes high resolution 3D shapes into a compact and continuous latent space an MLP is followed that decodes this feature into meshes once the uh trained um the vae model A diffusion Transformer is trained after it in the latent Tri space here is a visualization of the

44:20 generated meshes given a single image another limitation of this approaches is that they can only generate object Centric 3D difficult to deal with complex and expensive SC future directions include generation TOS General 3des in addable 3D and dynamic 3D or 4D in summary people aim to aim for fit forward 3D generation and improve the 3D generation to be faster more accurate editable and more General okay welcome back so uh so far we have discussed about the like a 3D object uh like trending with priors and with prior now let's move on to uh 3D scene Generations so first let's discuss what makes scene uh different from objects U these are some samples from um the alra verse data set and these are some scen from like a triple A and some open world games so I think the difference uh are quite obvious that for scen the complexities and the Travers abilities uh is is of a different level and also uh this is some

45:47 uh a clip that we showed uh earlier today about what about the masterb showcasing the compositionality so uh for the scene generation we have at least need to make it like composition aware and to achieve all of these and to uh make it uh feasible for like a training inference uh it also need to be Memory Invision uh no matter uh in terms of the representation we use or no matter in terms of the architecture that we choose okay so uh there are also uh some different perspectives to break down uh the SC Generations including uh data model usage uh this part is uh quite similar to uh the 3D object part and uh there are some uh different properties that each uh methods focusing on including some focus on the traversability and some focus on the compositionality and also the indoor and out SC are often uh handled separately and uh whether the representations uh is implicit or explicit okay so uh let's get started uh

47:01 first similar to the object uh training on 2D data and that start from uh the indoors SC and again s also similar to uh the object case uh this are these samples are uh traversible is which means that it is uh not that 3D but mostly still based on images uh but uh like 3D aware okay so the pioneering work is uh this GSN uh this the method is quite standard like create a 3D space sample a camera and render images then used again to Trend the model it has constraint that it requires continuous images and a corresponding camera process for training uh we can see from the results that most objects are already recognizable uh but of course the quality is not that good uh from today standard then the follow up uh followup work GDI adopt similar framework but train it with diffusion model it first optimize latent representation for rting SP and Camera process for each sample then they use diffusion model to learn to generate

48:15 these latent and the methods benefit from the advantages of diffusion models that is easy to add condition uh so it can achieve different kind of condition Generations uh like condition on the first image and to generate like different uh different trajectory or we can also condition on uh text prompt like the example go through the hallway or go the stair Etc and of course uh we have IND scene then we also have other efforts to the outo scene uh like this infinite natural zero uh strictly speaking it's uh iterative image predi Pipeline and does not guarantee any 3D properties it's a conditional generation pipeline that generates a sequence of images based on an initial image and a camera trajectory you can probably uh obviously tell the scene changes while uh camera moving uh the 3D consistency is uh not good and the followup uh work persistent nature further improve uh the 3D consistency at

49:24 least when the camera moving forward then backward or like spinning around the spot the output is more consistent but uh as the reposition is still not 3D so in inherently is difficult to be fully consistent in addition to traversability some work try to handle uh compositions I put an aster here because although the trending data is image it usually requires some 3D Ming b as additional prior which which means the underlying data is still actually 3D so this disc Co scene uh make use of the uh layout prior to achieve scene compositions with separated object generator and background generator and also scene discor and object discrimin as well so uh here are some result we can see that it can uh decompose some object in the scene okay next uh trending on 3D data there are many works on this subset the reason is quite obvious uh it's too difficult to collect SC level 3D data uh with reasonable

50:36 scale so uh here we introduce some uh an interesting projects that we did uh before the infinite City so this infinite City aims to generate arbitary scale CD data by combining 2D and 3D model the first stage they generate Tod category depths and normal maps of arbitary scale using the infinite pixel generation technique uh infinite again uh to be specific then it LIF uh these maps to sparse 3D vxo and perform uh 3D completions with ar base model uh for comput computational efficiency finally it performed uh it performed like muscle based neuro rendering with SC model uh the reason that no diffusion model was used is simply because at a time we did not uh when we did a project layer was not uh stable diffusion available yet the 3D structure is actually uh quite nice uh and the 2D rendering uh is okay but is of course not as good uh from today's from today's standard next uh finally what the the major Focus today

51:53 so the methods that use to the diffusion model as prior to achieve uh traversible scene uh Generations here's an overview of this branch of methods so uh usually we'll start from one image the image can be given or it can be generated by some like text to image model and then we have a we will obtain obtain the depth uh with the off shelf death estimator then we will leave it to whatever 3D station that we use uh then we will move the camera so we will sample a partial images and then we use let pret diffusion model to uh do the Imp paintings or out painting uh to get a complete images and then again with the death estimator to give the dep use Le information to update uh the uh the 3D representation so uh the following work Al L might have some different uh 3D presentation might have some different way of updating updating but the concept are uh the same so these text room uh using MH as the

53:08 representations so like we said uh sample camera impend uh use St estimator to G stth and then update the MH the scene looks decent and are indeed uh traversible uh but we also can observe from the mesh that there aren't too many details and many geometries are actually incorrect but as a pioneering work these text to 3D uh room results were quite amazing and this text Nerf use nerve as the representations the pipeline is similar uh have representations and move camera impant and then update the nerve so uh the quality actually pretty nice but obviously compared to text texture room the oppos scene is not traversible it only supports like minimal camera movement the recent Lucy dreamers adopts the most popular 3D representation gion Splat uh the rest are the same like impant Des estimate and then update we can clearly see that the visual uh visual result is improved tremendously however if you really

54:26 visualize them in the 3D engine and take a closer look the detail geometry is still uh very messy some work even incorporate a uh into the pipeline so uh this Wonder journey is a project like that I really like uh it also presented in this cvpr it adopts Point Cloud as representations and the pipeline is similar impant then update however let add language model into the pipeline to make use of the almighty knowledge prior uh first based on the scene descriptions uh memory it ask LM to create the descriptions uh for the next scene the description will then be used to guide the outting then they use vom to take both uh generat new SC and description as input and judge whether the generated images contain any unwanted effect effect so this can greatly help to stable the iterative generation process because the original issues of this kind of iterative process is that uh the arrrow will be accumulated throughout

55:38 the process so uh here are some uh sample result we can see that in this kind of like very imaginary uh scene uh we can generate like decent uh decent detail and the of decent scale then let also perform quite well in this kind of like real world scene and using the like real world images also uh they can uh do some uh cool application like condition on one image and then they can have uh like a like a different trajectory like a different uh different scene predictions here uh before going to the last part I want to introduce a work specializing at uh generating uh texture of a 3D scene so uh it is not feasible to use the same technique as a 3D object texturing due to the computational constraint here syntax introduced a multi-resolution texture field to encode positional features at different scale in the UV space the UV embedding are sle uh are mapped to an ALB image via a Coss attention texture uh texture decoder for

56:58 details please check it as the spotlight session so um the result is pretty good however as an optimization based method it takes hours to uh render a scene here's an examples of the collaborations of MH GPT that we intr uh that we introduced before and this syntax

57:32 okay so uh the last part how to achieve uh compositionality there are three major uh ways to uh to achieve composition uh to achieve compositionality so first uh the easiest way is to uh give it as a input or we can treat it as a learnable parameters or of course we can use as a prior to help us so first uh if the bonding boxes are given it is a simpl scenario we can use the conventional SDS to perform uh L Generations here are uh some

58:22 results next uh we can uh use the LS prior uh so there are many different ways uh to to leverage LS first uh this s with 3D they only use l to identify object of interest uh for example we give a scene give a prompt about about the bedroom you will you will uh you will answer you will El you know that the object of Interest usually uh include like a bed a chair uh or a table and uh the composition is learnable during the process here is an overview so they use the hybrid representations to handle uh the foreground and background uh separately and then uh the the object interest can either be user specify or use l as prior to help the process uh and we use the particle swarm optimizations uh to optimize the uh the the configur ations and the rest are similar so we use uh different we use several different prior like Panorama deps and perspective images and use as SDS to help uh the full scene

59:45 optimizations and here are some uh visual results of this methods so uh given a prompt uh we can generate uh 3D SC and the forr object can be manipulated and the details of a forr backgrounds are nice and of course it can support uh the scene with uh with different styles and of course there are diverse scene uh category including like outdoor scene uh or real realistic scene like imaginative scene Etc okay the next one uh list uh graph streamer as the LM to follow the S graph format of visual genome and generate a s graph based on input text then it DEC compose the S graph into like Global note wise edwise text descriptions and then optimize the SDF based objects in the 3DC using their corresponding test uh descriptions and here are some uh results uh showing there like a like a SC graph and a correspond output uh and this SCA 3D uh directly as the L to uh provide a set of 3D bonding box it is actually quite impressive like

1:01:12 the L while trend on uh like a text based uh data they have this kind of implicit uh 3D uh 3D understanding so uh the results are actually of pretty high quality as you can uh see from these samples uh also uh it it is more like a small 3D scene asset uh not really like a large traversible scene uh that we want for for uh like a larger larger use case like a V and setup okay so that's all for uh these three uh this 3D scene generation uh introductions and let's take a quick uh quick break and then uh let let's be back to the final part this is cyang from snap research and in this session I will talk about 4D generation and reconstruction before we start the tutorial I'd like to share a clip from Star Wars in this scene R2-D2 plays a hologram of a Jedi master likely Obi-Wan for many of us these Star Wars scenes with Holograms are embedded into our childhood memories and we would be incredibly excited to see such cool

1:02:31 technology could come into our real life and luckily science fiction is gradually becoming reality today we have VR headsets AR headsets mixed reality headsets and holographic displays that allow us to experience fascinating 3D and 4D content however creating 4D content is not trivial it's far more complex than just taking out a camera and shooting a video so in this tutorial we'll explore how we can employ generative AI to make 4D content creation more accessible first let's review what is already available for 4D generation very broadly we can consider physics simulation as a method for 4D generation Physics simulations help create realistic movements and interactions in a 3D space over time adding that extra time dimension of reality to our content also because of the huge interest in the gaming industry and the development of avatars for the metaverse current technology is already quite successful in creating human

1:03:32 animations this includes generating physically plausible motions and photorealistic facial expressions now that we have human engineered simulators and realistic human animations what's the next exciting research topic that we can do for 4D generation to consider this question let's look back in history and we can see a clear progression for 2D image gener we started with simple digit generation then moved on to category specific generation like face generation and finally in recent years we've achieved textto image generation capable of creating both artistic and photorealistic images similarly in 3D we began with methods that generated only the skeletons of objects then Advanced to generating a simple objects like Furnitures and ultimately achieve text to 3D object generation then in terms of 4D can we we follow the same logic and assume that we should aim to achieve generic 4D generation for anything for example

1:04:33 creating 4D content from any text prompts and this is the exciting Frontier which we're going to explore in this session to approach this problem let's first have an abstract view of what 4D generation is about at a very high level 4D generation can be oversimplified as a method capable of generating frames at different viewpoints and times for example imagine having a matrix of frames where the vertical axis represents changing viewpoints and the horizontal axis represents changing time steps this allows us to visualize how an object or scene evolves over time from multiple viewpoints and currently we have already made significant progress in generating videos from text which can be considered as generating one row of this Matrix and we have also made significant progress in generating static 3D objects which can be considered as generating a column in this Matrix but what we don't have

1:05:31 yet is a model that directly generates 4D content which means generating all these frames together in one go and this turns out to be a very challenging task if we consider the success of 2D generation it heavily relies on the scale of training data allowing the model to develop creative capabilities at scale for 3D generation progress is significantly behind 2D generation due to data sets being several orders of magnitude smaller now for 4D the situation becomes even more challenging one of the largest 4D data sets currently available has only a few thousand data samples this is significant smaller than 3D data sets and it is nowhere near the scale of 2D data so current research in 4D generation is structured around the limitation of having limited number of 4D data samples in this tutorial we will group these approaches into three types based on the portion of frames needed to be directly generated by a generative

1:06:32 model the first approach is very straightforward since we already have video generative models that can generate a video from a text prompt we can treat the problem as a 3D reconstruction Problem by lifting a 2d video into 3D in this way no new generative model is needed to be trained aside from the pre-trained video generation model but reconstructing 3D from a single video with Dynamic motions is challenging as shown in this animation it is mathematically an ill-posed problem because there are ambiguities between depth and scale since increasing the size of the object and moving it farther away from the camera can result in the same projected 2D image by shrinking the size of the object and moving it closer to the camera and given the ambiguity of between depth and scale of the object it is also not possible to determine whether an object is actually moving farther away from the camera or it is

1:07:29 shrinking in size and finally the problem becomes even more challenging when there is occlusions there is a branch in computer vision focused on 3D Reconstruction from a single video with Dynamic objects known as non-rigid structure from motion the first nrsfm method to my knowledge was developed over two decades ago in this work they tracked facial landmarks from video and reconstructed meshes representing the shape of the face they were also able to reconstruct simple structures such as the upper body of a giraffe a typical nrsfm method would take input of 2D Point correspondences created either by 2D Landmark detection or feature Point tracking the nrsfm algorithm is then expected to reconstruct the 3D location for each point this problem is ill posed so we need to introduce additional constraints to find meaningful Solutions common constraints include expecting the concatenation of 3D positions of points

1:08:30 to be either low rank or to have sparse structures following assumptions like sparsity and Union of subspaces and similar Concepts have been applied to denser reconstructions as shown here this nrsfm algorithm can reconstruct most regions of the dancing woman however you may notice that not every pixel is reconstructed especially in textureless areas like her hair this limitation arises because nrsfm is a two-stage method that requires obtaining 2D correspondences through Point tracking before performing reconstruction but reliable long-term tracking is very difficult in Real World videos for example there are inclusions and out of view which prevents you to have a complete 2D trajectory for every pixels and there a model drift problem where the tracker lost track of an object and drift to track other objects instead of the two-stage approach roach of tracking then reconstruction more recent methods adopt a different

1:09:29 Paradigm called analysis by synthesis this involves using a differentiable renderer to render a 4D representation and optimizing this representation to synthesize the input signals such as RGB images this single stage method is usually more robust than two-stage methods and since the optimization goal is to synthesize an image that closely matches real images analysis by synthesis methods typically provide High Fidelity results for view synthesis to perform analysis by synthesis we first need to discuss how to represent 4D to capture High Fidelity 3D shapes and motion next the target signal for the differentiable renderer isn't limited to just RGB images it can also include other modalities like depth maps from monocular depth estimators pixel correspondences such as Optical flows 2D Point tracking and semantic embeddings used to Lo localize object Parts in the next several Pages we will cover some

1:10:29 commonly used 4D representations and the current status of obtaining monocular depth estimation and correspondences and finally we will show that several state-of-the-arts 4D reconstruction methods are essentially a combination of these modules let's first talk about some commonly used 4D representations most widely used 4D representation can be factorized into two modules the first uses a 3D repres resentation to model a static template shape often referred to as the canonical 3D representation the second module is a deformation representation used to deform the template shape to synthesize Motion in recent years many papers have used neural Radiance Fields as the canonical 3D representation because Nurf can produce photorealistic renderings and it is relatively easier to optimize for complex shapes compared to traditional representations like meshes recently many 4D reconstruction methods

1:11:27 have switched to using 3D gaussian splatting this technique represents objects as a 3D Point Cloud where each point is a gaussian unlike Nerf which requires Ray tracing and point sampling gaussian splatting uses rasterization making it much cheaper to compute while still achieving similar Fidelity that's why more recent methods are adopting 3D gaussian splatting to deform the canonical 3D representation we need to discuss how to represent deformation commonly used deformation representations include neural networks that map a space-time point to its corresponding position on the canonical template shape another approach is using deformation Primitives like linear blend skinning which represents the motion of a point through a combination of rigid motions associated with different body parts or control points there are pros and cons to these two types of representations deformation fields are

1:12:25 more General and can theoretically fit complicated motions however due to their extreme flexibility they lack the inductive bias needed for accurate reconstruction especially with rapid motion on the other hand deformation Primitives like linear blend skinning are designed for articulated objects such as human or animal motions they are more restricted but tend to perform better when reconstructing rapid articulated motions up to this point state-of-the-art 4D reconstructions are achieved using 3D gaussian splatting with deformation as shown in this video this method 4D DS is faster to train and significantly faster to render compared to Nerf based representations such as uox while achieving similar if not better rendering Fidelity and because 3D gaussian splatting is an explicit 3D representation like a point Cloud we can easily control and manipulate different parts of the gaussian independently

1:13:25 this allows us to create interesting animations like this one where a Tesla Model X waves its doors now that we've covered 4D representations let's switch gears and discuss which signals can be used as the supervision signals for the analysis by synthesize one of the most important signal to use is depth map estimated from a monocular depth estimation network using depth maps estimated by a pre-trained depth estimation network is perhaps the most important component for robust 4D reconstruction this is because as we previously discussed reconstructing from monocular videos without any prior knowledge is not possible to disambiguate the depth of the object with its motion therefore we need to use external Knowledge from the monocular depth estimation Network to to provide direct supervisions to disambiguate the depth in motion monocular depth estimation itself was a very challenging problem a few years ago as shown in this

1:14:24 video video state-of-the-art method 3 years ago is not able to produce temporally consistent depth estimation and losses a lot of details papers like consistent depth of moving objects in videos is able to produce temporarily more consistent depth estimation by fine-tuning depth estimation Network specifically to a single input video but it still lacks sufficient details in the estimated depth map but recently things changed there has been great success in the community by by introducing Advanced self-training Techniques or utilizing video generative models to improve the accuracy of monocular depth estimation for example as shown in this video the recent version of depth anything is able to produce temporally consistent depth estimation with fine details another important puzzle for robust 4D reconstruction is how to obtain correspondences to supervise the deformation of the 4D representation the

1:15:23 most commonly used representation of correspondences are Optical flows Optical flows are typically obtained by having a neural network which takes two input images an iterative estimate the 2D per pixel motion between these two images in addition to estimating 2D pixel displacement between two frames recent methods like Cracker take a sequence of frames as input and can directly estimate longterm 2D trajectories here are some 2D tracking results from Cracker it is able to track points on the moving foreground objects quite accurately across a long period of time more recently spatial tracker extends cracker to not only track changes in 2D pixel locations but also output changes in depth allowing it to generate 3D trajectories this is a very promising Direction and may become a key component of future state-of-the-art 4D reconstruction methods another type of Correspondence is specifically trained for humans and

1:16:23 animals this method extends from dense human pose estimation the the idea is to learn a continuous surface embedding for each point on the template shape and also train a neural network to predict the embedding map from a 2D image with the predicted surface embedding map correspondences between the template and image can be found by nearest neighbor feature matching continuous surface embedding has been successfully applied for reconstructing humans and animals now we have discussed 4D represent ation and different modalities that can be used as supervision signal for analysis by synthesis now let's review a few state-of-the-arts 4D reconstruction methods the first method is Bano which is able to reconstruct articulated objects like cats and dogs from videos the recipe for banm MO is to represent 4D by using Nerf as the canonical 3D representation and uses linear blend skinning as the deformation in addition to RGB images it

1:17:24 is supervised by Optical flows segmentation masks and also continuous surface embeddings a more recent method mosa has been developed for reconstructing generic Dynamic scenes it is currently one of the best methods for achieving highquality reconstruction for novel view synthesis as shown in this example it successfully reconstructs the bb-8 robot rolling forward the recipe for Moa is using GS dual querian blend skinning as its 4D representations and it is initialized in super rised using monocular depth maps Optical flows and 2D Point tracking on the project page of mosa they also showcase impressive reconstruction results from videos generated by Sora despite these encouraging results it's important to remember that reconstructing 4D from a single video is still a very challenging task current 4D reconstruction methods are still fragile and far from production ready an interesting study by

1:18:23 this paper die check found that many methods are evaluated on data sets with relatively large camera motions and slow scene motions this makes the input videos more like multiview videos or what we call effective multiv view as opposed to strict monocular videos which are casually captured with smaller camera movements and faster scene motions testing all recent state-of-the-art methods including mosa on casually captured videos reveals noticeable artifacts for example there are unwanted jittering and artifacts around fast moving fingers even for relatively simpler problems like object Centric reconstruction state-of-the-art methods like Bano still produce results with low Fidelity for instance in this example the head and tail of the dog are not properly reconstructed and if you see closely you can see that there is a hole at the intersection between the body and tail this brings us to the

1:19:20 second approach using 3D generative priors to provide stronger supervision for 4D reconstruction and generation the idea is this given a generated reference video with object motions we can also condition on one frame from the reference video we can condition on one frame from the reference video and use a pre-trained 3D or multiv view generator to create the 3D Shape for that frame this gives us one row and one column of our frame Matrix the challenge then becomes how to generate the rest of the frames one straightforward idea is to use the remaining frames from the reference video as conditional frames and feed them into a pre-trained 3D or multi- view image generator to generate each column however this approach is problematic because each column is generated independently without considering temporal consistency as a result the generated frames may not align smoothly over time as shown in this video the generated 4D

1:20:20 content Jitters when viewed from a fixed Viewpoint so how do we solve this problem the most intuitive solution is to incorporate temporal models or video generation models when generating the remaining frames however pre-trained video and multi viw generative models are not designed to work together to jointly generate all these frames in this situation there are two potential Solutions one option is to train a joint generative model to directly generate all frames in the array but this faces issues like the lack of 4D training data which we will discuss in later slides another approach which does not require any training is to use techniques like score distillation sampling to generate these frames specifically to perform score distillation sampling we first need a 4D representation which we optimize so that its rendered frames at different viewpoints and times match the pre-learned distribution of videos and

1:21:17 multi viw images here's how it works at each training step we sample a random row at add gaussian noise to the frames and feed them into a pre-trained video diffusion model this model produces denoised estimations and we evaluate the SDS loss applying this rowwise score distillation helps enforce temporal consistency preventing Jitter in the rendered videos from different viewpoints in addition to rowwise SDS for temporal consistency we also perform SDS for each column using 3D generative models like pre-trained multiv view image generation models this ensures that at any given time images rendered from different viewpoints correspond to the same reference image with the same motion finally it's crucial to apply SDS to both rows and columns failing to do so can result in multi-perspective Illusions which are not desirable many 4D generation methods follow the score distillation sampling approach the first

1:22:17 text to 4D generation method aav 3D uses score distillation sampling to learn a 4D Nerf with hex plane representations 4df which will be presented at this conference improves upon math 3D with higher resolutions and finer details more recently many Works have switched to gaussian splatting for example dream in 4D can generate various kinds of moving objects however there is still a lot of room for improvement as you can see here these results are object Centric with simple backgrounds and do not feature more than one Dynamic subject per scene more crucially if you look closely at the generated videos they are not very photorealistic and have a strong cartoonish Style with unrealistic shading improving these aspects especially achieving photo realism would be very beneficial to improve this we first need to understand the source of these cartoonish Styles the issue mainly comes from the data used to train current 3D

1:23:18 generative models the most widely used 3D data set obverse consists of a few million 3D assets assets but unfortunately many of these assets are not realistically textured or rendered as a result the models trained on this data do not produce realistic style 3D assets one promising way to enhance photo realism is to use video generative models video models can be jointly trained with both object Centric 3D data and a large number of real videos as shown here video generation models like the snap video model can generate freeze Time videos with circular camera motion and static foreground objects these objects are placed realistically within background environments with realistic shading and other details this motivates our recent work called for real which to our knowledge is the first to generate near photorealistic 4D scenes with detailed backgrounds as we will show later it can also produce multiple

1:24:15 Dynamic objects in one scene so how did we achieve this the first step is to generate two videos using textto video generation models first we create a reference video with a relatively static camera trajectory but with object motions then we generate a second freeze Time video conditioned on the first reference video with a circular camera motion and almost No Object motions given these generated reference videos and the freeze Time video and with the help of score distillation sampling we can reconstruct a canonical 3D representation using gaussian splatting and also capture its temporal deformations the optimized deformable gaussian splatting allows us to render freeze Time videos at different times covering almost 180° in terms of field of view it also enables us to render stabilized videos from different viewpoints because we use video models we can generate not only object Centric

1:25:15 scenes but also multiple Dynamic scenes with very detailed backgrounds here are more examples demonstrating that this approach can generate very realistic appearances and motions even with complicated backgrounds like drum sets and complex lighting effects there is still plenty of room for improvement such as achieving higher resolutions longer and larger motions and wider view coverage a major limitation which I want to emphasize is speed unfortunately methods based on score distillation sampling require a lot of computation and are very slow to generate even short 4D scenes for example 4 defi takes more than 10 hours to generate a scene our work for real intentionally reduces the number of SDS iterations but it still takes roughly 1 and 1 half hours to generate a scene this is a significant limitation running on high-end server-based gpus for over an hour is Not Practical for making the technology

1:26:13 accessible to the masses we need to find a more promising approach with higher inference speed this leads us to the third strategy for generating 4D content the the idea is to have a generative model that can directly generate all frames in The Matrix as a multi view video without any timec consuming runtime optimization steps with these generated multiview videos you can reconstruct 4D representations using relatively simpler and more efficient methods this all sounds promising but what would this multi viiew video generator look like to understand this let's first review how videos or multiv view images are generated with diffusion mod models at a high level the noised frames are fed into a sequence of Transformer blocks inside each Transformer block the frames first go through a cross attention layer which applies conditions to the latent features then they go through a self attention layer Which models the spatial

1:27:12 and temporal relationships between patches of the video frames this sequence of cross attention and self attention is applied multiple times until an output layer produces d noised predictions to extend this architecture for generating multi view videos we can make the following straightforward modifications first we can duplicate the cross attention and self attention layers along the temporal Dimension which is like running multiple multi view image Generations in parallel however as discussed earlier this would still result in jittery outputs since temporal relationships are not modeled in this architecture a straightforward solution is to add crossframe attention layers in addition to the self attention layers these crossframe attentions help to reason about the temporal motions in monocular videos from a fixed Viewpoint this essentially summarizes the most important component of recent multiview

1:28:10 video generation Works while the design of the network architecture is somewhat straightforward training such a model remains challenging due to the limited scale of available 4D data sets there has has been some success in training models to generate category specific objects like humans thanks to a few large scale human motion capture data sets that consist of multi viw videos for training for example the recent work human 4dt trains a DIY Transformer that is able to produce decent results of humans from different viewpoints in times but for Tex to 4D generation without category specific constraints there aren't many large scale 4D data sets available this means training every parameter in this large architecture while maintaining good generalization is not feasible Therefore several recent papers choose to freeze the backbone Network and not learn the cross attention and self attention layers instead they only

1:29:11 modify the crossframe attention layers to improve temporal consistency between each stream of videos at different viewpoints one way to design the crossframe attention is without any learning for example it could simply linearly mix the keys and values of the image patch embeddings across time this simple strategy has been proposed by stag 4D which shows some encouraging results in generating 4D objects another promising approach is to learn the crossframe attention layers from limited 4D data a recent paper called for diffusion which appeared on arxiv just a few days ago does this by using around 900 animated 3D shapes from the obverse data set the results show show higher quality compared to other methods and it will be very interesting to see if further improvements can be made when trained with a larger number of animated 3D data from obverse and finally let's quickly recap what we've covered in this tutorial

1:30:10 we've discussed three main approaches to achieving 4D generation first generating a reference video and then performing 4D reconstruction second using score distillation sampling with multiv View and video generative mod models to create 4D content and third building a 4D multiview video generator currently research and text to 4D generation is still in its early stages and there is much work to be done we can aim to develop more robust 4D reconstruction methods preferably with a feedforward architecture alternatively we can focus on training larger scale multiview video generators building larger 4D data sets and developing more advanced training algorithms I am personally very excited about this area of 4D generation and can't wait to see exciting future progress from the community and that concludes my presentation thanks for watching

Questions about this episode

Who speaks in "23718 3D 4D Generation and Modeling with Generative Priors"?

Shingly (Host), P. Jong (Host).

What is discussed in "23718 3D 4D Generation and Modeling with Generative Priors"?

In order: 0:00 4D Generation, Nerf, Shingly, Snap Research; 20:00 P. Jong; 30:00 Score Distillation Sampling; 1:00:00 4D Generation, Snap Research; 1:10:00 Depth Estimation, 3D Gaussian Splatting, Nerf, Mosa; 1:20:00 Score Distillation Sampling, 4D Generation; 1:30:00 4D Generation, Score Distillation Sampling.

What is mentioned in this ComputerVisionFoundation Videos episode?

12 people, works and ideas are mentioned, including 4D Generation, Score Distillation Sampling, 3D Gaussian Splatting, Nerf, Shingly, P. Jong. Each one on this page links to its own page with every episode that mentions it.

How long is "23718 3D 4D Generation and Modeling with Generative Priors"?

The episode runs 1 h 31 min; the transcript is about 11,760 words in 77 timestamped sections.

More from ComputerVisionFoundation Videos

About this transcript: the episode's spoken words as captioned on YouTube, split into timestamped sections. Reclip.Me locates every person, work and idea in the text, keeps the time it is first named, and quotes the words around it verbatim. Last updated January 23, 2026.