23722 End to End Autonomy A New Era of Self Driving

What is discussed, and when

1. 0:00 – 15:00: Hong Kong University
2. 15:00 – 30:00: Gaia
3. 2:00:00 – 2:15:00: Dreamer
4. 2:15:00 – 2:30:00: Dreamer, Gaia, Drive Dreamer
5. 2:30:00 – 2:45:00: Dreamer

Notable quotes from "23722 End to End Autonomy A New Era of Self Driving"

• AI some radical experience from the industrial and from the Academia we also have home Le assistant professor from Hong Kong University and research cenes at the open J Lab we'll talk about some critical point about the foundation models to.

  about Hong Kong University · at 1:27

• And the pedestrians and the complex reflections of the puddles on the water that prism starts to really come into its own it's able to you know actually reconstruct those…

  about Prism · at 17:18

• Stronger policy I think immediately after that dreamer B1 came out that was 2019 and Dre we want introduce this rssm architecture which is effectively a recurr net with some fancy distributions.

  about Dreamer · at 2:11:07

• Far and of course I mean hasn't been claimed as a world model but this year we also Sora coming out as kind of impressive video generator and with sort we started seeing the…

  about Sora · at 2:17:32

• That we wanted so we decided to reink what we were doing and that's how we started working on Gaia so Gaia is our way of rethinking scaling word models with data and compute and the what's gu guy is a word model based on an auto regressive.

  about Gaia · at 2:21:55

• For that long in terms of generation so that was about like a year ago so after that we started seeing a lot more work coming out in word modeling for autonomous driving this is just one of these examples and we started seeing that.

  about Wave · at 2:25:19

• Multiple cameras see seven eight nine multiple sensors and they're very high dimensional so dve War model I think is one of the first model that came out I started…

  about DVE · at 2:27:18

• Into the world of word modeling and started to tackle high resolution video generation so this model Vista it's quite impressive video generation at a very high resolution so…

  about Vista · at 2:31:08

Who and what is mentioned in this episode?

Companies & orgs

• Wave — Company · Business focus · 5 mentions
  Wave is a company pioneering end-to-end autonomous driving systems, mentioned multiple times as a leader in the field.
• Hong Kong University — Company · Affiliation · 1 mention · first at 1:34
  Hong Kong University is mentioned as an affiliation of a speaker.

Products & tech

• Prism — Product · Product focus · 4 mentions
  Prism is a self-supervised 40 parametric scene representation tool developed by Wave for simulating dynamic scenes.
• Gaia — Product · Product focus · 3 mentions · first at 21:32
  Gaia is a generative AI for autonomy, a generative world model developed by Wave.
• Sora — Product · Product focus · 2 mentions
  Sora is mentioned in the context of video generation technologies.
• Dreamer — Product · Product focus · 9 mentions · first at 2:11:12
  Dreamer is mentioned as a product in the context of autonomous driving technologies.
• DVE — Product · Product focus · 2 mentions
  DVE is mentioned as a product in the context of autonomous driving technologies.
• Mile — Product · Product focus · 2 mentions
  Mile is mentioned as a product in the context of autonomous driving technologies.
• Vista — Product · Product focus · 2 mentions
  Vista is mentioned as a product in the context of autonomous driving technologies.
• Drive Dreamer — Product · Product focus · 1 mention · first at 2:25:40
  Drive Dreamer is mentioned as a product in the context of autonomous driving technologies.
  Examples and we started seeing that there's been a little bit of a shift towards like Laten future so Drive dreamer is one of these examples so Drive dreamer is like Li…

Full transcript with timestamps

0:08 right seem like we have lots of people today thank you so much for coming so um I'm long and my resarch SST at B firstly I'm going to give a introduction of this tutorial about our motivation and the schedule so our title is into an autonomy a new era of self driving so our motivation is that last year we saw that both industry and Academia are shifting towards and do solution especially one Milestone is a Tesla sfd beta V claiming that they're using a single into neuron Network to change their driving model and another Milestone is uh the uni ad paper one the s b paper and here here's also a very good surve paper paper about the undu adops driving paper if you would like to have a look so for today's schedule we really want to give us like amazing tutorial about inent driving so we have lots of sessions today so firstly uh we will have a foundational session so we have jimy Shelton the chief scientist one

1:24 wave uh we talk about the RO to embody AI some radical experience from the industrial and from the Academia we also have home Le assistant professor from Hong Kong University and research cenes at the open J Lab we'll talk about some critical point uh about the foundation models to solomy uh we will have a 20 minutes copy breaking between the next session is a deep dive sessions um Bor senen at wave talking about lots of hand down experence uh Niki will talk about the neuros simulator and januka will talk about the world model and uh we will have another coffee break for 1550 minutes and then lastly olac will talk about the language me driving so lastly we have alahi uh the head of research AI at W uh we'll talk about the some of the future direction of and2 and the PS driving so uh without further Ado please allow me to introduce our first speaker Jimmy Shon uh Jimmy is a chief scientist

2:35 at V uh leading the science group building foundational model for embodied AI to enable safe and adaptable atomus vehicle PR to this he was a partner director of science at Microsoft and head of mixed reality AI Labs very his shaped foundational features such as body tracking and hand and eye tracking that enables connect and Hol lens too let's welcome

3:17 Jimmy okay fantastic well thank you all for joining and it's amazing to see such a a full room and uh uh excited we're all excited to be here uh to talk to you today I'm I'm gonna give you a little bit of a sort of overview of of um of this space um uh that hopefully will point the direction for some of the talks that are coming later on set the scene um and and inspire you by you know the the the opportunities ahead uh in the space of end to end uh autonomous driving and the applications uh for uh of that towards uh embodied AI more generally so you can't have failed to see the accelerating progress in AI over the last few decades uh you know breakthrough after breakthrough covering all sorts of domains uh from from handwriting language speech computer vision of course um uh have have been you know falling uh year after year at an accelerating pace and uh I I thoroughly believe that that driving is

4:20 is next on this on this list but driving is is really challenging you're out there in the real world surrounded by you know safety critical scenarios uh here is just one uh image from uh the Streets of London that we drive every day this is not an edge case this is a a very normal occurrence um your system that you build has to be able to deal with that um and uh the complexity of the real world the the the challenges the the um the variability and and the the fact that there are humans very unpredictable humans uh that populate that environment but if we can unlock uh uh the potential here for AI to come out of uh you know the the cognitive domain the the chat gbt domain the the the generative AI domain and actually embody that in the real world the opportunities are immense and of course we've all seen uh these these kind of examples from science fiction over over the years um but that's we're really on the cusp of that

5:26 science fiction becoming reality a real for us in our day-to-day lives uh and I think as we'll see in this talk I hope uh driving is probably the first place this is going to really uh mature into uh into mainstream use for for everyday life across the planet now looking back at the the last uh 10 15 years maybe um there's been a clear Trend uh from uh you know major advances uh around the sort of image data set and Alex net on top of that and everything that's come uh from there in in the space of the computer vision Community here of course um leading rapidly on to the amazing advances in NLP uh with the Advent of Transformers and uh you know uh the the incredible uh capabilities of next token prediction uh in terms of language and other forms of generation that we've seen um and really the next Frontier is going to be taking these advances and putting them out in the real world the open world the

6:32 unconstrained unstructured uh complexity of that where safety is Paramount so just to give you another example of the the kinds of environments that we are operating in every day uh in London here is a again not an edge case this is a pretty normal thing uh very uh complex environment with uh Road works here you've got passing traffic uh we've got to negotiate out into uh complex uh traffic that is again very unpredictable um navigate through this and we're able to do this all without uh HD Maps um and just pure endtoend intelligence so you know what's the difference what is n2n and what why is it different so if we look back to uh a more traditional uh autonomous uh vehicle sack it looks a little bit like this so you've got some sensors um typically it's a case of throw as many sensors as you can at it because they all of course add value in some sense uh pass that into a perception uh a set of

7:39 perception algorithms computer vision algorithms detecting bounding boxes Lanes uh objects Etc um simultaneously try and localize yourself to a a map often a very high definition map that you've scanned in great detail um and then pass that on to uh you know a planning system that s of predicts what what's going to happen next with the other actors in the world and then a policy that decides how to act given all of that information and pass that onto a control system that finally actuates it now this can work right there are fantastic proof points of this in the world um and uh you know it it does work but uh it comes with a very high cost and nobody has yet done this uh in a way in a in a way that is really a sustainable business um at scale um and it's it's it's very very challenging to scale for a number of reasons um you know the the the perception sack you have to supervise it you have to label a lot of data that's

8:40 expensive um uh you know furthermore you know as a human driver imagine you could get into a vehicle and put on the world's perfect a perfect head mounted display that turned your camera sensors your your eyes into a perception so it just visualize the bounding boxes that you get from the perspect perception algorithms and everything else right would you be comfortable to drive from that not being able to see the real world I wouldn't that's a nice thought experiment that I think like motivates the fact that you you're throwing away stuff right you don't know exactly what it is but you're throwing away stuff it's an information bottleneck uh next of course right this this diagram up the top is quite complex it's uh it's there's quite a lot of boxes there and the interplay between them is uh non-obvious so even if each individual one you can you can manage and understand quite well this is a big

9:32 complicated system those Maps they're also expensive to build and uh you know the the more sensors you throw at this problem the more compute you need the the harder it is to integrate into a vehicle um Etc so instead what we've been pioneering at w for the last seven eight years now uh is really an end to end system and uh it it looks a lot simpler so similar kind of sensors on the left or we're very flexible with what sensors we use genuinely an endtoend network uh pixels in on one side and uh driving trajectories out of the other side and then a control system um on top of that and this has you know really many Advantage advantages in addition to the the overcoming the problems that I I talked about a moment ago with the traditional sa um it comes with benefits like it it's computationally homog homogeneous uh very easy deploy to different silicon uh on the vehicles um we can generalize through

10:34 data and we've been doing this the last couple of years as we've been taking what was able to drive just in London uh on just one vehicle to driving on many vehicles across the UK and now across uh the Us and other places as well and that all happens through data it's pretty scalable and an economic solution to this um you can use just cameras if you want to if you want to throw in other sensors you can um and we don't need those high definition Maps as well um and and It ultimately it works and it worked better um uh the performance and safety of this U outperforms what we've seen from handc gooded Solutions so here's a few interesting examples of of the kind of emergent generalization you get when you put this on the road so you know this top left example um it's learned to uh to pull out of the way and it could see that vehicle coming uh and these are again typical narrow streets in London where

11:31 you've got to negotiate with with users it learns that there's someone coming there's a gap let's pull in there wait for them and then pull out not something we've programmed something that it learned uh similar similar example here where it sees that this van is uh indicating to pull out and uh perhaps too politely let pulls back and uh lets it uh turn around uh a slightly more unusual case in the bottom left here with this uh this car that's decided to reverse up the road um uh and find a parking spot uh but again effortlessly handled with these um and maybe a more mundane example there on the bottom right but just some flutter on the road and again we didn't have to program any of that in it just it sort of learns that flter is to be avoided so uh avoid it uh we've been working as I mentioned on generalizing uh the driving across different vehicles um we now have it integrated into about six different

12:28 vehicle types um and this little video on the right is is the same neural network driving two uh very different uh vehicle uh platforms one following the other as it as they drive around the Streets of London so I'm going to spend uh the majority of the time today talking about um these three topics which I think are at the frontier of of uh a embodied AI research simulation multimodality and Foundation models and you know again this will hopefully be a a very brief introduction to some of these areas that you'll see more about uh in the the talks that follow uh the rest of the afternoon so without further Ado let's dive into simulation and of course if you're doing endtoend driving well there's no you know obvious way to sort of simulate bits of it you have to simulate the whole thing now that comes with challenges but it also comes with opportunities um and I'm going to show you some of the latest the latest uh

13:28 work that we've been doing on endtoend simulation so here's a few you know of the various simulators out there uh already today um lots of um fantastic work from across the the community and across the industry um and they all kind of in a sense do different things right they all attack the problem slightly differently from different uh perspectives and with different requirements and that's that's perfectly reasonable um what I think is uh you know really important for a simulator are these things on on the left here right you've got to close the gap uh it's not just a visual perception problem as we said it's a behavioral problem so you know you can't just have photo realism you have to have behavioral realism in your Simulator the world is complex it's not just a bunch of rigid bounding boxes moving around in a scene graph uh relative to a static world the world's more complicated that that you have

14:24 pedestrians you have cyclists the formable objects the the lighting incredibly challenging often the textures of those objects change the brake lights the the wipers the indicators the traffic lights um it's it's a really rich world out there and a lot of the uh you know it's very easy to make seemingly naive seemingly okay uh assumptions but then they they really limit you in terms of what you can reconstruct um you've of course got to not just simulate the the visual side of things but the whole interaction the whole robot um interacting with the world it's got to be scalable and and ultimately scalable really most of the time implies data driven um and last but not least you need some control over this you need to be able to especially around the long tail simulate uh important safety critical scenarios and I think we're getting that so we'll we'll talk a little bit about some of the the work we've been doing at

15:20 wave but is ticking a lot of these boxes so um we've taken the approach of of res simulating the world so capturing the world from real cameras as you see in the top left and then um turning that into some form of of simulation resimulation of that of that scene and um you know it's been amazing to watch the progress in your rendering uh over the last few years that have enabled us to even imagine doing this kind of thing on top of that sort of uh you know reconstruction technology we've got to turn it into a simulator and so what we we've been building is something called ghost gem um which just turns this into a full Clos Loop simulator so you take actions the the driving model takes actions in the simulator that updates the state you get get a new set of images coming back in and you turn that Loop and uh you you see what happens um here's a few examples of the kinds of uh scenes that you can reconstruct uh there

16:22 uh uh on the on the right but uh what I'm really excited to to talk to you about today is something we just launched yesterday so sort of really hot off the press which is prism one um and again this really comes back to the critical importance of simulating dynamic and deformable um actors in the scene they the the most vulnerable Road users in many respects um and yet so easy to to to just sort of gloss over because they're hard to simulate uh so prism is a self-supervised 40 uh parametric scene representation um that essentially just tries to reconstruct everything it sees uh over time and give us some control over that so A couple of examples here uh in London and the Bay Area um but you know it's when you get scenes like this with these uh cyclists uh and the pedestrians and the complex reflections of the puddles on the water uh that prism starts to to really come into its own it's able to you know

17:27 actually reconstruct those pedestrians as they're pedaling and it's not just some kind of rigid cartoonish uh uh cyclist or pedestrian that that you're seeing here oops um here's another example where we're deliberately now perturbing the camera path to with this sinusoid to show we've got that that level of control here this is a simulator and and I mean some really remarkable things here I think first of all the the fact that we're getting this pedestrian at all with their umbrella um and then then all of the complex lighting and reflections of the uh the water on the road there uh furthermore the the traffic light you see um up there changes color as part of this uh this simulation here's another example where we're doing some more nuia synthesis um it it pauses here and hands to the left and right um and you know again this is it's really important to be able to do this because if you've got a closed loop

18:26 simulator it's going to drift off the the path that you Grove originally and so it's important to be able to uh to see what you know to let that simulator uh play out in these examples instead of you know letting the the the car keep driving we've sort of Frozen the the camera and we're moving time backwards and forwards um and so you can see that the dynamic objects in the scene are again they're moving we're able to uh to reconstruct those um in in beautiful uh accuracy and furthermore you know again things like the traffic lights which uh you know would be hard to approach in other ways um and a couple more examples here with challenging lighting conditions Reflections on the road the uh the glint and the the building up there um uh subtle things that but they they do matter when it comes to uh testing the uh the the the the driving models that you're You're Building uh to a level of

19:31 fidelity that you need now as part of developing prism uh we uh have also been of course using uh a internal data set uh to Benchmark this and to track progress and we're delighted to also share um this data set it's called wave scenes 101 um you can uh download it from our website and this is designed specifically to help the community to focus on problem 4D scen rid construction um it's super important I'm sure there are uh you know you you all have amazing ideas in the audience here uh of how to uh improve this even further uh and we hope to sort of Empower and excite you to to do that um it's got uh you know variety of scenes different countries uh different Road uh types weather layout uh weather lighting conditions lots of cyclists and pedestrians um and it we we've even specified a a heldout camera so we can test nuos synthesis um in this uh particular uh in a a nice way uh just three examples relatively

20:38 simple ones in these cases but showing the variety of different lighting conditions we've got here from Sunny to to rainy to to dark um but perhaps more interestingly we've got some really really challenging scenes in here so this top one uh Haw back to the first slide in the talk uh with all these pedestrians uh Crossing the road and again you know a true Simulator for solving autonomous driving is going to have to deal with these kind of uh situations but of course you want to go beyond um just uh re simulating exactly what happened um and uh to be able to create new things as well um and uh you know here both you know we can combine the neural rendering and Genera of AI to start to uh bring this up and even and simulate these these rare edge cases so uh this time last year we we launched Gaia um which is a generative AI for autonomy a generative World model and essentially what that means is You

21:43 can predict uh the uh you can generate videos but you can controllably generate videos so you can action condition it on what can come next or indeed language condition it to to tell it uh you know what the weather should be like or you know what objects should be present and you know this this is uh you know architecturally a fairly straightforward next token prediction plus some video diffusion on the on the end um but um uh but as you'll see s of as you'll see in a second come came with some really amazing results of course this has become a really exciting and crowded field over the last couple of years now um and delighted to see progress further progress in this uh space uh in the years to come but uh here's a few examples of of gu just being left to its own devices just let it let it generate don't try and control it or anything and I I think the beautiful thing about this is these

22:42 are these are long form videos it's generating these for for long periods at a time um is generating a variety of different Road layouts conditions uh it's able to deal with the dynamic agents it's got some clear sort of mental represent of of dynamic agents versus the static scene um and they're certainly not perfect and there's rough edges around them but on the whole it it's pretty uh it looks pretty convincing you can do some nice things with this and and one of my favorite is uh producing diverse Futures so you give it the same starting point uh and then you let it you know diverge and so in this example on the left the the white car in its its imagination has has reversed and let us pass but the white car on the right has decided to to pull forward um and we then give way as the ego vehicle we can also control it though and this is where it starts to get back to simulation how do we uh you know uh

23:42 start to close the loop here and um in this example we deliberately steer the ego vehicle uh towards the oncoming traffic and lo and behold Gia imagines that this oncoming traffic swerves out of the way to avoid us well recently we've been looking at how to use guia to uh add objects and and and uh join the dots between what we saw with ghost Jim and prism and and what we've just seen with with guia uh and in particular one one thing that's looking really exciting and promising is is the idea of of you know just adding objects uh realistically so here we're taking a a reconstructed scene uh from Prism uh and drawing this tuid uh of where we want a vehicle to be and then letting guia fill it in and lo and behold a car appears with realistic lighting realistic Shadows um uh Etc a couple more examples of a bus and two more cars here and then here's a video example where all of the dynamic agents here now

24:47 are being uh inpainted by uh by guia as it goes and there's some really interesting remarkable stuff especially around the fact that there's uh you know this is multi view generation and we're able to uh you know accurately uh infill this this object in both cameras uh simultaneously and then uh you know back to prism I think you know another really interesting topic here is is counterfactual simulation so taking something that really happened and then changing it in some way so here we're taking this pedestrian removing that pedestrian and then you know you can compare a versus B what your driving model does in these situations um and this is a really really clean clear and I think causal demonstration therefore of what matters to that driving model so I expect to see much more uh around counterfactual uh simulation in the in the months to come okay the uh moving on then the next topic I wanted to touch on briefly is

25:53 multimodality and um really this is all about language uh and language meets driving um which sounded completely crazy when we started this a couple of years ago but I think hopefully probably does not sound quite so crazy uh anymore um opening up all sorts of applications from explainability uh to the you know improving the raw intelligence of the driving um and building trust and uh safety around uh autonomous driving and um you know the the there are so many uh exciting applications here but uh the first one I want to talk about is lingo one um which was a grounded visual uh question answering um uh model that we put out a few months ago um and here you know relatively simple architecture again you know uh put in a snippet of your video uh into some encoders uh into a large language model that's been uh fine-tuned and then turn that into a dialogue Loop um and you can we we sort of explored this in two two modes one just

26:57 generating common country as you see on the left so uh here we're driving along it's saying I'm maintaining my speed I'm now I'm slowing down due to a jwalker I'm slowing down in preparation for the upcoming left turn and on the right you've got a true uh vqa system uh where you can ask what it's seeing and it will describe uh that and you can keep you know as a as you can keep refining and asking more questions uh as to what's going on in the scene uh we extended that with lingo showand tell which uh now you can ask it things like what are you paying attention to and not only will it uh tell you but it will also you know draw with sort of a semantic segmentation type output overlaying onto uh the the the video to to to Really sort of connect what it's seeing with what it's saying but more recently we've started to extend lingo uh and released lingo 2 which is all about um not just commentary and

28:01 explainability but actually driving at the same time um so I believe this was the the first V uh language and driving model that went out on on public roads actually controlling the vehicle as well as being able to uh explain its actions and so a few examples um here accelerating uh increasing speed making a right turn to stay on the route reducing my speed for the upcoming turn turning right to follow the route Etc and this is the same model that is producing the tax is also actually the policy and producing the driving commands here keeping left to follow the route turning left to follow the route another example reducing speed for a turn turning right to stay on the planned loot increasing speed limit to meet the speed limit reducing speed due to J Walkers moving past a park vehicle uh keeping speed the road ahead is clear there's Crossing is clear um keeping speed the road is clear we can also now join the dots

29:07 between the the language models that are driving that we just saw with lingo 2 and the simulation technology and use simulation to explore controlling uh the the driving system with language um and so in this example we've got a a simulated environment where we ask it to stop behind the bus on the left and instead to overtake the bus um on the right and lo and behold this language based prompting changes uh what happens with the driving as You' expect and uh I think uh the the the next thing here is is back to those counterfactuals changing things so here we're taking uh an original video and using guia to insert these uh these vehicles so that's the only thing we change and then we look at both how does the driving plan change um and that's these sort of Wiggly lines that are probably a bit hard to see on the the slide um but also how the text change and so on the top left it says increasing speed to match the speed

30:06 limit on the bottom left it says slowing down to match the lead vehicle speed um and on the right stopping youe to the red light uh and stopping uh behind the stationary traffic so again it's this you change one thing in the scene and you explore what happens and if you can do that uh uh successfully over the statistical sample you're getting some really good confidence about what's going on in the brain of these uh these models the final thing I wanted to uh touch on today is around foundation models for embodied AI now a foundation model as we uh all we've all heard them the term bandied about I think this is a pretty decent definition it's a model that's trained on a diverse set of data that can be adapted to a wide range of Downstream tasks and of course huge huge progress uh in the community on this the last few years certainly starting with with language and then Lang language and and vision

31:09 coming together um uh major advances on this slide of course more recently uh in the space of embodied AI uh this has also been garnering attention um and and sort of my favorites are the these ones at the bottom uh and especially the ones where where you're starting to see cross embodiment results where you can train across multiple different robotic embodiment uh jointly and get better drive better policies for each of those envir those those embodiments then had you train specifically for that uh embodiment on its own I think that's an amazing result and I think really points to the Future here where we can start to learn from uh you know uh from vehicles uh and uh from humanoid robots and and any other kind of uh robotic uh embodiment and take the data and learnings from from all of those and combine those together into something that's bigger than the the individual um parts of this now when it comes to

32:13 Foundation modelss in embod there are some genuine hard challenges that are distinct from the the more traditional cognitive AI Foundation models we've seen so far data being perhaps the the primary one here the data just doesn't exist on the web right it's not like you can go and scrape uh you know crawl crawl the web's uh uh web pages for text or craw YouTube uh or equivalent for for videos that you just it's not there so uh you have to find it somewhere now turns out in in autonomous driving that's relatively straightforward compared to some other forms of En environment you've got all of these cars now all almost all modern cars are equipped with loads of cameras there's dash cameras out there Etc so I think autonomous driving is where you can actually solve this um but it is going to be challenging nonetheless um once you've got all that data of course that's a lot of data and you've got to process it and that leads

33:08 to huge compute demands for training uh and finally you got to deploy this on a robot uh of some description and robots as you may know if you've worked with them are don't always do what you ask them to do um you've got the challenge of distilling a big foundation model down onto something that can uh run uh at low latency on an embedded ship you've got that highly ambiguous uh uncontrolled operating environment you've got to work in um and of course you've got to deal with the the safety and and and validation of this but as I say in the space of driving I think we can make some real real progress here um you can get internet scale video data uh you can get a subset of that that's uh slightly smaller which is more around uh embodied video uh you can get large scale driving data from deployments uh of of existing vehicles around the world and then you can get you don't need much on top of that the icing on the cake is

34:12 high quality driving data that you actually learn a policy from and you know I think it's it's really interesting to look at you know this kind of diverse sources of data and and question you know is it useful is this useful and I think the answer is yes I think you know each of these videos has something interesting to tell us about the world how the world Works how humans work how physics Works um and so I do think that there's a lot of value to be had by learning from a very diverse set of uh of video data and other forms of data to help us uh even if all we care about at the end of the day is is the driving Tas and so ultimately I think these multimodal Foundation models um are going to help us unlock uh safe and generalizable autonomous driving and hopefully much more Beyond so to wrap up uh today thank you for for joining us for this tutorial um uh I do think embodied AI is the next

35:18 Frontier uh in uh in Ai and the self-driving is the way to to to get there first to to to show the value to the world of embodied AI we've seen how uh advances like prism and Gaia are helping us to simulate complex Dynamic scenes uh end to end um we've seen how language uh can help us with explainability and building trust uh and understanding of uh how these endtoend systems work uh and how uh embodied AI Foundation models are going to be one of the key levers to extend this further uh so with that I will uh I will wrap up thank you again for coming uh you'll you'll see us around there's a lot of us from wave here this week we've got a booth on the floor um we've got the rest of the tutorial this afternoon of course um and uh please come talk to me talk to one other team that would love to uh to engage and continue this conversation thank

36:24 you good thank you so they have a few minutes for question so U yeah we want to so you uh did explain abouta and how you can create scenarios right how do you go from say creating new scenarios where you can plug in stuff to creating the actual tale scenarios that we care about for example something falling off a back of a truck or something is that coming from user prompts that we write today or is that auto also automated how do we think of yeah so I I think so the question is how do you go beyond just sort of inserting vehicles to generating more complex scenarios I think that's one of the the the big challenges in simulation right is it's not just doing it I think that that could be done by hand relatively straightforwardly but the question is how do you do it scalably how do you how do you do that um I'm not going to claim that I have all the answers to that today but I do think that's that's really the the Crux

37:23 of this this challenge is is how do you keep pushing the the envelope there in a in a sustainable scalable way you can generate and realistic data why don't data Foundation model yeah so the question is um we can generate synthetic data why do we use those um rather than than than um a models I'm not sure the answer is that you don't is one or the other I think you can certainly think about both and I think when you think about I think it's especially around safety critical scenarios they're not going to be things that you're going to be able to capture much of yourself now there may be dash cam footage and and and in quite some volume on the web that does capture those kind of uh safety critical scenarios but even then I think there are you know there are very good reasons why you might want to simulate those more more um programmatically um I think you also very quickly get into the sort of

38:33 interesting almost philosophical debate about uh you know is it better for a model to learn from the entire uh web versus uh you know uh rather what happens when the web is entirely simulated it's synthetic itself right when chat GPT is spat out almost every website on there can you still learn from that can you still get value from that um I think there are some there are some arguments both directions but yeah great talk I had a question about do you think like Gaia and prism are kind of becoming a virtuous cycle in that way that first you get a real word scenario and then you use prism to simulate millions of those scenarios and then go back to GA so it's it feels to me that they're like kind of becoming one thing yeah I think there's definitely some convergence between guy and prism um I think exactly where that takes us we will see um but uh yeah you can you could certainly imagine taking a

39:32 fully generated guia scene and then res simulating that and that giving you more levels of control um ice said pH is back so you really showed some great uh capabilities fromo I'm interested where in your like production pipeline from development testing validation deployment vehicle testing valuation where do those tools playin and where do playin okay uh question is basically where do all of these things that I talked about today show up in our production stack I'm not going to talk about our production Stacks I'll say that up front um but uh the the mission of of the having the reason we have a science team at way that we can invest in these kind of um you know fundamental research breakthroughs is precisely because we want to take that into production and we want that to impact our sack we want that to you know accelerate the progress that the company is making so I I won't answer with any specifics uh for obvious

40:44 reasons but um you know absolutely we are pushing these as fast as we can into production is there any systematic way to evaluate the Genera War it resembles cortically ah yeah can you can you evaluate the realism of the of the generated world I think that's a really interesting question I don't know maybe J will talk about it a little bit later on but um I think there's a more General version of that is how do you evaluate the realism of a simulator um and I think that you know there are lots of there are lots of approaches um uh all of them imperfect the ones I've come across so far um but all add a little bit of value so you can look at you know the realism of the the the generated scenes you can look at how Faithfully it able to recreate an existing scene if it's a fully generated one though it becomes a bit more challenging but then you know you could start to look at distributions of

41:42 behavior in that simulator um things like that question from the safety perspective how would you anticipate or wish to see safety research change to be able to support this kind of EV what would you say to n when convin their their perception of what it takes to get this yeah I I think um so what would I say to the naysayers um well I think well part of it would come back to that um thing I that that thought experiment I did earlier which is would would you put on an hmd and drive your car around the real world only seeing bounding boxes um I think that's part of it I think part of it is that you know you can get the the feeling of security from testing each individual box in that complex thing but when it comes to the whole you still have to test the hole as a system um and that's no different to what we we have here um I I think and and then I think the other thing is just the the mounting evidence

42:52 that actually it does work better in practice and so the the because of that I think we just have to you know that doesn't mean we've solved all the problems around safety by any means but we need to go and invest in that and we need the community here to to engage in that process and to be you know to be part of that and and to you know put that sort of skepticism of end to end aside and think okay this is happening it's real it's working how do we now go and and build that safety assurance that we need last question

43:35 uh it's not broadcasted so uh prob we will have recording released later we'll have recording released later that's for the right so is this um let me welcome our second speaker hang leang is a assistant professor at the University of Hong Kong and our research stist at open dri La shanai his research focus is on adopt driving and embody AI he well known for leading the ENT adop adopt driving project uni ad that won the C best paper award in 2023 and the birth ey view perception work by former that W the top 100 AI papers in 2022 let's welome home thank you okay uh hello everyone uh my name is H from Shanghai I just arrived in the St so there's my be D jet lag and so my my title uh the title of my talk is uh could Foundation models really resolve uh end to end of driving so how many guys of you uh vote for this uh so Foundation model will definitely resolve uh end to andony okay uh really resolve or partial uh can

45:11 resolve for the time being down okay uh I see a few hands up only a few okay and uh uh how many of you have already uh come from the industry uh many hands and uh the from the Academia okay okay only a few from the Academia okay that would be great um because I'll I'll I'll briefly uh talk about the uh uh the road map in the research because uh we are told I'm told I'm asked to uh give you a deep dive or work through of all the recent endtoend Auto driving uh research uh especially in the uh Academia so my talk uh consists of three parts uh so the first one is a introduction to the end to endend and the data sets the evaluation and also the motivation and the second part is some uh Recon work uh from from from the community not only from our group and the last part is a few challenges and closing remarks as to uh what's the so-called future of end to end aut driving uh because I just came

46:23 back uh uh from a few test trials uh especially a a China a Chinese corporation uh called Huawei and another is from Tesla and I also uh test a few other uh products I mean we just see there and to test from like the uh from from The Source destination and to the destination so I think most the cases in the a driving nowadays are well resolved so the only reason that we are here is that whether we can mitigate the the marginal gap between uh the resoled part with that of the poter cas part so the first one is a uh introduction uh this is the problem setup uh as we all know uh Auto driving consists of three parts where the perception is about recognize the objects whatever in whichever form 2D Bond marks or 3D Bond marks and the prediction part is given the uh uh eego vehicle so how could you predict the intent or trajectory uh in the environments of the other uh uh agents and the third part is uh planning where

47:40 you have given all the uh proceeding uh information uh whatever is perception results or uh prediction results and you're are going to plan your own trajectory uh in the next uh few frames so we do have lots of challenges so uh the first thing first as Jimmy already mentioned I suppose the definition of end to end in our case uh in today's talk is that this is a suite of fully differentiable programs that takes raw sensor input uh as input raw data I mean whatever camera light canas data uh into the system and then produce a plan or lowlevel control actions as output which is shown in the uh block here so compared to the conventional modu based design uh where each module or functionality would uh output the uh direct uh results in whichever form uh you can name it uh mapping RS or Bonnie marks uh so in the end to endend Paradigm so each module would fit forward the uh features or representations and you only got uh one

48:55 or you only got major objective functions or training laws to get back propagated through the uh standard uh deep learning optimization process so this is a uh main the major distinction from the end to endend compared to the uh Moder based design so there are also some uh preliminaries uh the first one is uh data set and evaluation so as I mentioned earlier we are tasked to do the pr uh planning uh task which is to uh perform a few we points or trajectories of your own eego vehicle in the next few frames so there are some uh renowned data sets uh the first one is newsin and uh I think all of you no Nuance right so the behavior or interaction I mean the uh behavior of the surrounding agents with the interaction with your own eego vehicle is realistic uh it is captured from the realistic data set and the strategy here means um once the data set is collected you cannot uh modify the trajectories right uh for

50:06 example from from point A to point B as long as the data set is collected uh so the trajectory is fixed so this is called the open loop evaluation and for the matric which means uh the matric is to uh evaluate uh how well you did the uh trajectory prediction or other T is the uh L2 errow and uh I think the other the rest data say like vimo or Aros is also very famous and it is performed in the similar way and recently uh we see the new plan data at Benchmark uh it consists a way larger a at least the one magnitude of scale uh of the uh length of driving scenarios compared to the previous new weo and Aus and it is machine learning based uh corrected and it is Clos Loop so by closer Loop uh here we see so the Nexus surrounding environment is dependent on the current uh prediction output of your own algorithm so uh in term of of uh evaluation it got a various uh uh mck whatever you you can uh evaluate in the

51:20 average display error or in the uh cion rate or the uh a comprehensive score called PDM score uh so basically it is a uh weighty sum of uh five or six uh evaluation metrics so for the details you can resort to uh this uh aome beh have uh rle and uh the rest part is a finatic generated data set uh so to name a field I think the most uh famous two is uh dve SC but unfortunately I never saw a technical report or some uh open source rapple from from dsim so over the years it's a lot of demo and I could admit it's very uh fantastic uh fantastic but we from the Academia we we only uh uh see the technical report as the uh solid evidence so that's why we are using the kala which is all open source and uh it can be uh it can generate unlimited data because of simulation right you can manipulate all the enir M as assets and it's handcrafted uh handcrafted and is open uh uh it's open looped and is a

52:34 close uh uh uh sorry it's uh open source and it's uh evaluated in a Clos Loop so uh in the color it has a driving score which consists of two parts uh the first one is uh how complete are your are your whole route uh finished uh in the in a given uh in town uh multiplied by the uh infraction panalty uh which if you collide with somebody else or with the uh environment uh so you're going to be punished so this is the data set so why do we need a uh end to endend so over the years uh I've been giving some talks as for the reasons and uh as I prepared last night uh I see the most of straightforward uh motivation on this is the uh um Global optimization ability which means this is example from a work called uh unad and uh since each module is connected by the features not the direct results so even though here the uh red circle so this is the perception results where you turn in this case uh

53:49 you will turn right but the car doesn't see there is a obstacle behind the tree so even if the perception fails and we can see during the uh feature uh attention attention feature layers uh in the planning part it can also uh regress all the attention part so my my my takeaway is that uh in this endtoend uh Paradigm or Paradigm Shift uh you can recover from the uh previous modules failure uh even if the perception is inferior and you didn't recognize uh the objects but compared to the conventional part which is a modular design uh you cannot even ever do that right so I think this is the most important motivation for us to do uh the end to end and another thing is the emergence of foundation models uh which is a scam all so we have a massive amount of data and uh you got as assuming uh you have a lot of gpus or infrastructure and you can show or demonstrate strong generalization capability but to be

55:01 honest I haven't heard any uh or few solid uh uh evidence to claim that we are doing the uh Foundation model for Auto driving so I think most of the audience came from the industry uh I must assume you have a lot of gpus or data so please uh if you have these resources you can share some public results uh your technical B and uh so what we are seeing here all the way here is that uh I think we are here uh I mean the green dash line here is the uh endtoend approach but it hasn't shown very uh promising results or capabilities and as I mentioned earlier if you ever uh C some trial uh Drive uh in this daily like Tesla FSD in their vehicle I think most the cases are well well resed so why we are sitting here or why we are uh uh doing research especially in Academia for Auto driving so all the way here we are resolving this part or this Gap so to mitigate uh what the non-end to end system cannot do

56:12 and to uh ex ex exhibit great generalization given the uh faster development from the uh Foundation models are essal and this part is for the efficiency uh so in my previous talk I think uh efficiency is the uh uh good part but I really think this is not especially you are deploying the large Foundation models uh so the FPS of foundation models is below 10 but uh as I know from the industry this requirement is as for 30 or even higher and the only uh downside part I think for the end to endend of driving is lack of interpretability which I think this problem resumes in all computer vision domains uh this is because due to the endtoend new network you cannot explicitly explain uh which part uh takes the major responsibility and someone may argue okay uh you have the unfair evaluation as to the overloop or other part uh as for the uh simulation but I think those are the challenges those are the C the

57:21 challenges that can be solved whatever from the Academia or industry so those are the disadvantage that inherently resides in the end to end Paradigm so uh given the uh limited time uh how many time I um so uh this is a classic uh algorithm uh it's called the uh trans Fielder and uh you can see uh here I will just skip that um uh how many times I left and where set okay and it's nowhere okay uh so I I I'll dive into this part um so this is the un8 uh Paradigm and I think uh okay and here this is a a little bit uh credit here I I must to see that given the limited time so all the way back uh to the endtoend system uh days back to this project called open pilot so I think uh this is a startup company uh in San Diego uh I see there CQ are very active on Twitter and we really bought their equipment and tested in Shanghai and this back to two years ago I think this is the first time that drives me to

58:37 see uh Auto driving can be such a simple thing to use a simple Network to directly output the control signals so uh uh if you have some effort or have uh more interest I would strong strongly uh encourage you to to to uh buy some product from them and test it in your own car okay this is a personal advertisement for this uh okay and uh uh the the the second part is uh research Panorama uh which is uh uh I I have skied some uh conventional uh approaches the introduction of them so the very original end to endend can be dated back even into before the 20s uh where I think this comes from some block from Nvidia or I'm not sure uh so the input is even block image and relying only on the neuron Network which is just three MLPs and you can uh have a control system and then I here I omit uh uh a lot of uh related works here uh before the uh 2020 so here uh so the y axis is is a goal uh I prefer to use the word

59:58 goal uh uh you can uh refer to it as performance or if you are from the industry it indicates the MPI if you from the Academia it indicates some performance score on some specific uh Benchmark and recently we see the UN and many others uh for example V I really like V work is is is way efficient than the uh unad so the gap or the uh uh G from the previous work I think to my opinion is a formulation which which Swift from the model based design to the uh M2M and uh there is no scale of data I mean the uh size of this circle indicates uh the the uh data scale because they are all doing on the uh new SC well study new SC and uh we all see from the industry there is a hype that you can utilize a different End to End Auto driving uh system uh at at least in China I think literally every car company claims they have uh deploy the end to end system I'm not sure whether this is the case in Europe or other the states uh so for

1:01:08 example I've seen the advertisement from uh Mobile in CS this year and also other Chinese corporations just to name a few and uh I think uh there are some concurrent work uh which is Gia uh and prism uh which is uh released uh yesterday a few days ago and also the J or lingo other work so uh to be honest I think the marginal Gap is uh the the G the gain is really marginal because uh I think the uh data scale helps but V or alarms uh should do better um uh I could talk about this topic for the whole day but given the time uh I should first put the uh conclusion conclusion here so given on the future so what's the future should we uh uh fully leverage the all the data given from the large amount uh test scenarios from the simulation or the So-Cal embodied AGI so what should we do so my takeaway or missing ingredients uh consist of uh four parts uh the first one is the data everyone knows about the data right and

1:02:22 the second part is this is where the university fact faculty or research students steps in so we can still do a lot of uh technical contributions in architecture whatever you are using the to dual system or you are using the word model or using the uh video prediction such as uh what they do in the Sora and the last missing pieces pieces is the info of platform I'm not sure if any of you or most of you have dealt with uh training with hundreds gpus uh if you are doing this you must be uh much remember that the training St stability is very important and also the deployment How Could You deploy the uh uh uh uh large scale Foundation models uh on board uh I'll been asked a thousand of times even though I came from the Academia so I I think that's a major responsibility uh from the industry so let take our uh responsive uh respective role okay uh so my take on the generalization on the generalization to achieve the m2n

1:03:36 of the driving system starts with the data so we have the uh we we we how to collect the uh massive amount of data and then we have to think about the effective um action or policy learning how could you uh utilize the uh whatever reinforcement learning or imitation learning uh here are some examples of previous work uh from our group or from the uh other groups uh including the MC3 or un or travel adapter they are mainly focused on the uh algorithm part and also uh as the uh emergence of llms we got interpretable CA of reasoning uh whatever you are utilizing the llms uh to interpret the the scenarios but for now for the time being as of today uh I see no uh evident Improvement uh given from the large language models so one of the reasons is probably that um the output of the conventional test in AO driving whatever is planning or motion prediction requires numeric or exact trajectory output uh which is uh for the

1:04:48 LMS is incapable at all to Output the exact numeric numbers so we will see a lot of of d uh outs in the driving scenarios right and most 99 of them are got rejected by the CPR by mother other conferences uh so I I think the major uh uh the major takeaway is that uh yeah the LM doesn't Express the uh very good results or generalization as shown in the other domains such as the General Vision domain uh as a as it shows so then we got the uh closer Loop simulation uh so uh closer Loop simulation is really important because you you have to interact with the environment right uh so the trajectory is not fixed as long as the uh algorithm uh has some output the environment has to shift uh according to your algorithm so in such case the end to endend Auto driving can be deployed with zero Gap to the production otherwise it's only a prototype St and then we got uh once you have the co simulation you have the demo

1:06:06 collection where you can collect a lot of safety critical scenarios and you can uh design reward or other feedback depending on your algorithm uh design uh whichever is imitation learning or uh RL and then it comes to the CT deployment okay so I'll just uh uh work through a few uh works if you are taking it seriously uh there is a very good survey paper uh written by uh by D I'm not sure if he's here or not uh so we have summarized all the detailed work and group them into them into together and we draw the cola uh driving score here and uh uh so if you are seriously uh want to uh work on the end to end Al I will highly recommended so uh I'll also skip this part uh I I will be a little bit lengthy uh here so uh we have a we have a poster uh on on on Thursday I'm not sure so this part is uh as I said earlier right um we are in the age of foundation models so how could you utilize all the

1:07:20 YouTube data or how could you uh Express gr generalization ability so this is the motivation of the j8 work where we utilize all the uh uh internet data from the uh YouTube driving uh cases so as you guys can see here uh those are all the driver uh videos uh in the front wheel uh so uh it it has seen almost all the globe uh in all the cont continents so um and using this Paradigm uh you can uh you can for sure using the self-supervised learning or unsupervised learning to see the potential of creating a foundation model in aut driving yourself and see how it performs in the downam task such as planning and the other work is um I skip that we we really got a lot of the slides uh and uh the the next uh part is Visa project so I really like uh this project uh is uh recently released on archive and uh so given the Vista work the the the Gen work uh we have limited data on new things or we have the uh representation

1:08:42 capability which has a very small scale of resolution in uh Al driving and also the control flexibility which means um previous work using only uh single modality or in compatible with planning algorithms so in this work we have uh built on the G data set which consists of more than nearly 2,000 hours of driving data and then we have uh the resolution is much higher compared to previous uh Jad uh dve G or other uh works and also uh we have more control flexibility but yet again I think this is a very marginal work because we are still facing the problem okay you have the foundation model for driving then what so how can you apply the foundation model how can you drive like with like a human driver like we are using these Foundation models not to mention the deployment thing how can you drive smoothly and uh uh uh in some uh of nowhere cases or in some uh col cases uh you can generalize or uh right this is a

1:10:01 cap core capability of foundation models but for the time being here uh I haven't seen any work uh that uh has expained such capability so most of work just a uh sof here which generates the very fancy uh Sor version in the driving cases so this is where uh our University students or research Scholars uh have could invest L of time to to work out so so this slides show a few bit of Visa um which exhibits High fality uh in the future prediction or continuous long Horizon goout uh I knew this is nothing new given the uh emergence of SAR but in the driving at least in the driving domain I never seen such high resolution or long Horizon I I think it's a 50 seconds or 20 seconds I'm not sure sure um uh but again given the uh uh amazing performance of SAR uh someone say this is no big big deal but uh please show your work okay and uh uh this this is the Z shot uh action controlability uh I'll skip this part is just a uh

1:11:17 more uh better inform performance okay so this is the last part I'm sorry for for for the L par and uh this part I really want to see the challenges or closing uh remarks uh of what's the future or what's next in the uh Auto driving because I'm now this I'm Al often asked uh does it make sense to do uh research in the University so uh my guess is um so if you are doing the perception or prediction on conventional uh benchmarks no f is is is probably that uh I mean what's the big deal of improving 2% of MEAP on new things uh let the task be resolved by the computation domains I mean there is a whole bunch of people doing object detection in this whole cvpr right the reason of all of you come here to this home is to resolve real world problems for AO driving we are a applied science uh domain right so there I really think there there's no big deal to to work on this so please stop doing some research on using but no hoger

1:12:33 hoger we are very close friends uh he just visit us in Shanghai but uh no offense to new at all I mean this is a great work um okay and uh previous typ and prev Tye indicates last year we got the end to end uh uh tring topics everyone talks about it and nowadays we have the new baby cut uh word model dual system uh and other NE rending techniques right so for the Academia uh so I don't have too much results is there any impact research left so my my opinion is uh maybe uh in occupancy or in mapping or in motion uh prediction I don't know uh uh maybe there is um but for the industry uh we often heard uh we have lots of gpus millions of driving data uh but we can get cannot get access to it so could we uh uh beat Tesla or other winning corporations no offense to other corporations and Achieve L4 once4 using n2n So my answer is definitely not and because this is because uh there are just so many unresolve challenges so if

1:13:51 there is only one page of takeaway uh in this talk today it's going to be next one uh this one so I have list several dimensions in terms of data so we need high quality or large scale uh data so uh this is just a rough uh division of uh research and Engineering uh I'm not leading to either part okay so for the Academia I think we need some research especially use in your rendering whatever go is planing uh we can see lots of uh amazing work from recent cvpr or C to generate high fality or controllable simulation as the word engine to interact with the environment so we see some counterparts in previous year so the very exampler is color but it doesn't do well right everyone knows that but I haven't seen any uh well popular uh platform or open source project than color so I think doing this part is really impactful and also for the engineering part uh everyone knows that is a scalability collection so

1:14:59 recently I heard this book and I have checked some chap cap chapters in this book is called human in the loop uh machine learning uh I would strongly recommend you guys uh to have a look so for now I would say we would in order to achieve the foundation model we need at least 10,000 of hours uh given your reference new is only five hours and uh 80% % of them as just the straight going straight scenarios okay so do not report any MEAP on new things to me or to some conference again we just have too many archive papers each month right and for this SAR is okay that's good for that and we can see that uh for I just take this photo from the Twitter uh this is a reference on the FSD so this is uh uh the per billion of miles so after the emergence of V12 which utilize the M2M according to that right um so their their data just uh blooms like exp exponentially so we really need a lot of data uh uh to achieve this but this is

1:16:14 not a responsibility for one single Corporation but instead it does need a reunion from across countries or across um many many efforts from different parties right um so the second part is algorithm or methodology part I could also uh elaborate or explain in many hours so for like uh given the word model um or video generation we can do so these are the techniques or the tools this is why I put in a smaller font so so nowadays many people just say word model is the only solution to to all of us but in my opinion it's just a tool and the the the red part is the goal or the task we need to achieve the closer Loop feedback or to achieve the long Horizon planning and also for the engineering part is the efficient which is the deployment which is the forever topic that we are going to discuss and to achieve this the tool should be uh the Dual system as two system one system two or model compression or

1:17:25 whatever um novelty is going to be there in the Pro module and also some people might see in the future is it solely based on the end to end or solely based on the uh uh modular based Des I would say it's a it's not a dichotomy it's it's not a yes or no I would say it's an explicit design with um with explicit design what we we have uh uh something like in the UN ad where you have the perception but it's still uh in end system okay so for this I would strongly recommend you could resort to this uh IM mod AI survey paper uh I rarely recommend papers nowadays but this is a really good paper that you can uh reference to uh uh I draw many uh insights from this paper I think this is from the Google deine team but another team um okay and the last part is the computer and info uh to surprisingly I I see just a few discussions in this conference to discuss about it especially we are doing the auto driving

1:18:39 research if you're doing the foundation model I think there are just you need many okay uh for the University or or not and this is not indicating the whole group I mean for one project so if you are doing the J or Prisma I'm not sure Prisma use even not sure this is a secret or not uh but for me I think if you are seriously doing the foundation model Pro driving at least 50 uh or so gpus per project is needed and if you are in the staying in the industry I think at least 1,000 um um yes I think so and the theability or how to guarantee FAS IO or other hardware issues are also Al very critical okay so we we we we have switched the research Paradigm so uh a few years ago uh people might think you you might only focus on the novelty or algorithm technical contribution but as the large Foundation models I think all uh all the people in this room would agree that data and also the infrastructure also matters way L way uh

1:19:52 uh better than the uh technical uh technical part so uh again this is a very uh brief challenges and for more details I will strong strongly recommend the survey paper uh that we wrote A Few uh months ago uh so it it has more details on that and finally the task or the goal should be the same uh whichever you are from the Academia or the industry that is to chieve the driving Comfort or experience considered uh uh at least consider in your research okay so this is U my opinion on that so the final of the final is to how to achieve this AGI goal this uh uh climbing to the Mars uh you utilizing lots of tools whenever you are using mm or with models or whatever so data as word engine as I mentioned earlier and algorithm you can do a whole bunch of things uh here uh whatever is on each topic and uh info uh and platform and uh deployment I think those are all very important uh ingredients uh uh to

1:21:04 achieve the end to end on driving so with all that uh thanks for my team and a lot of guys have just arrived in this CPR and uh they are all the author of the uh retrospective work and you can see hi to um them okay that's all for my talk thank

1:21:31 you think we have time for maybe one or two questions I see the first time was there at the back great timing to because in engineering there is supp the complexity of one If You observe a in the system you can

1:22:11 trace yes uh repeat the question uh so the question is uh how can you uh interpret or track all the buxs uh given some uh system uh in end to end uh sense so unfortunately this is uh the very true uh downside part of the design disadvantage part of the end end to end of driving because um if you are uh using a model based design you can easily track uh which parts the recognition or the prediction whatever so this is where so there is a workr for now is to have a exp explicit design where you have uh intermediate losses or intermediate outputs and when uh the planning or uh the vehicle suddenly paries with the road and you can see the intermediate results of the perception results from the network to see uh whether or not it FS or not but it's still an intermediate solution for tracking the uh bugs okay one more question and then yes yeah thanks for the nice presentation um if we look at the future of the endtoend

1:23:33 solution and let's say it is a a system of networks or differentiable methods and if you read then zoom in on the first part the perception part yeah do you think in the future computer vision tasks such as uh a video bopic segmentation of other dense prediction tasks will they still be relevant as a as a method to initialize that perception module or as a form of deam supervision or in the future we don't need these computer vision tasks anymore uh I would not go that extreme okay but my B my best on this is uh yes do not do segmentation or tracking on driving again because open or other expertise from their part can do it very well and then you can utilize the vrm to F tune on the aut driving uh I mean the segmentation whatever the Tas right so all that research at cvpr is not that uh relevant at Le not yeah especially given the large scale of data and you can verify the scaling law is actual thing

1:24:39 right it's just my personal view right don't do not propagate that oh sounds sounds s okay sir all right thank you very much let's thank our speakers and we have a 20 minute break we'll be back at quarter past 3 so please come back for the next not um so yeah thanks for joining today my name is nikil I am a lead applied scientist at wave um focusing mainly on offline valuation using um tools like ghm to sort of um validate our driving system system and today I'm going to talk about how um we use neural simulators at wave and also how we're looking at measuring progress um in this field cool so just a quick overview of the agenda today um I'm going to talk briefly about how we are using neural simulators at wave and why they're needed in the AV 2.0 field um and then I want to talk a little bit about how we're thinking about uh benchmarking environment creation for neural simulators um as well as talk a little

1:25:58 bit about the new data set that we're releasing to um help uh Inspire progress in the field um and then some concluding remarks um cool so why do we need um a Clos Loop Simulator for right so um I'm sure if you guys have been to wave presentations before you've seen this slide many of times um but this slide highlights um the differences between um the classical AV 1.0 approach um and a2.0 um so if we look at the AV 1.0 approach um it's sort of a very modular stack that where we go from have a pipelined approach of sensors in perception planning um and control um as opposed to this if we look at AB 2.0 um we have our sensor in put coming in and uh we we have one neural network that is optimized to do both perception and planning in uh parallel um so this is a new way of thinking about driving and instead of trying to solve it um with robotics we try to develop a system that mirrors the way a human thinks about

1:27:15 driving so um given this um what are the implications for how we actually design our simulators to accommodate this um so if we look at AV 1.0 um and if we look at sort of validating the two big components in stack being perception and planning um we can start with perception on the left so in a perception system there's no sort of decision- making component in the system so we can actually utilize open loop data to validate the accuracy of our perception systems since there's no decision-making component there's no domain gap between offline and online um predictions and so this is a very effective way to Benchmark perception systems um and on the right if we look at the sort of um how we validate planning systems in AV 1.0 um since our planning stack since the planning stack um operates on top of the outputs of the perception system um we can actually do this Clos Loop um sorry we can do this closed loop

1:28:19 um simulation directly in the perception of space so what this means is the closed loop uh simulation engine uh can actually be a much more it's a much simpler task given you don't have to actually synthesize noral views directly in sensor space um whereas if we look at AV 2.0 and given we're doing perception and planning in parallel um we need to actually for to develop a Clos Loop simulator we actually have to um synthesize noal views directly in sensor space um and on top of this not only do we have to uh sort of render things directly in censor space but they actually have to be very photo realistic and the domain Gap to the real um world has to be low um yeah so if we then look at how we actually um would use or how we think about development of Av 2.0 um one of the big desirable profer is when we observe a disengagement in the real world we don't want to make the same mistake twice so here's an example

1:29:31 where we have a real world disengagement um where we have the model um where we have an empty zebra Crossing um on in in central London and we can see that the driving model in control sort of unnecessarily slowed down um at this empty Crossing which is not desirable um given you can sort of continue without continue at your current speed so given um given this disengagement now we really want to make sure that we never have the same disengagement again um and we want to be able to sort of prove this with our offline systems so if we start to think about I mean how we can actually go about doing this what are the options we actually have here um so the first option um is open loop verification um here um we're just using the log um the log sensor data from The Real World and running it through next iteration of our driving model to measure how it would perform in this in this l in these log States so the really

1:30:39 good thing about open loop verification is the domain Gap is zero given you're actually using the real images to um run through your new model so it's very um the predictions are actually very indicative of what would be the prediction these states in the real world um one of the difficult things with open loop verification is because it's open loop you can't actually um see the outcomes of the decisions that the model made so what this means is you only really get directional signal about how the model is improving offline so if we look for example at this video on the left um we have a new model being tested um in this in these log States um and the way points are in orange whereas the original model is in blue and we can see that the new model is predicting a much more consistent speed going through the zebra c um but still given this how can we actually know if this sort of prediction

1:31:41 was sufficient um to avoid a disengagement in the N it's very difficult to do even as humans because it's we don't actually know what the next prediction at the new state would have been um but with that being said um open loop verification is very cheap to run and we can scale this to thousands if not millions of scenarios and at this scale the directional Improvement can start to really provide valuable insights when you have millions of these tests running and you're getting directional signal on each one of these tests you can actually start to make um uh sorry you can actually start to provide scores um for the new model and control um so after open loop verification another option we have is sort of creating a handd designed digital twin similar to the um engine like Cara um the main advantage of a system like this over open loop is that it's closed loop so you can actually see the

1:32:49 outcomes of the decisions that the model made um but even with expert uh technical artists generally the domain Gap remains quite High to the real world so it's very difficult to make the jump and say um if this if we have a handd designed digital twin of the scenario it's very difficult to make the leap and say that this disengagement wouldn't have occurred in the real world as well um and once we actually move to the resimulation domain this burden on technical artists actually increases even more because they not only have to sort of create realistic looking assets but they actually have to match um the distribution or the assets we saw in a particular scenario um and so scaling these kinds of systems to every disengagement you see and uh want to create unit test with is really difficult um since the main bottleneck here is the human effort to create the environments and yeah now if you look at

1:33:52 third option is using a neural simulator which we all love um so if we look at the uh the properties of a neural simulator uh it's also closed loop so you can actually see you can actually allow the model to um change the state transition from away from the logged uh images so you can actually start to move away from just observing um decisions and move towards looking at what outcomes would have been um a neural simulator is a data driven uses a data driven environment creation um which because of how the optimization of the environment is done it is um directly trying to match the real world distribution of the scenario wec um so what this means is there's no requirement for a technical artist or a human in the loop we can uh pass video data in and get uh 3D or 4D simulation out um and this makes it super uh suitable for the resimulation use case where we actually want to um take the scenario we

1:35:02 saw on the road uh replay it offline and allow the new model to actually um control the sort of uh actually have control of the state transition um so now if we look back at the original example we started off with so again we see on the left that we have this real world disengagement of the mod SL down um and on the right we can see the same model we saw before with the orange um way points we now see it being tested in ghost um where we can see the speed stays consistent through the zebra Crossing and doesn't have this dipping Behavior so we can actually now see that the outcome of these predictions compounded and we can make more uh we can with more confidence make statements and say that um the new model wouldn't have also had this disengagement and for us this is huge um because being able to do this offline um I mean when you go in the real world you'll never see the same situation twice and so you need

1:36:07 simulation to actually make um comparisons between two models on the same scenario um and we're seeing a shift in this direction throughout the industry not just the Wade um where um UNIS from Wabi new ad just a few examples but we're starting to see the whole industry move in this direction of utilizing data driven techniques for simulation um yeah and now I mean one of the big aspects of a neural simulator is the environment that's used um to actually represent the world in the simulation um earlier this week we announced prism one one of our to reconstruct 4D scenes um directly from video data um and it's used in our in Ghost Gym arnual simulator to actually um synthesize uh the environment such that a new driving model can be driven um offline so today I want to look a little bit deeper into the environment generation and how we can um start to measure our progress towards creating better environments

1:37:24 um so yeah how do we actually measure the quality of these reconstructions so first and foremost we got to stay uh keep things simple we got to have image based metrics so these are things like photometric reconstruction quality metrics pick your favorite metric dsnr L Pips ssim um they're all super useful in being able to actually um Benchmark and that's the quality of how our um environment uh how our simulated environment matches the real distribution um in addition to this we can use geometric cues such as depth sface normals um Etc to actually make sure the um environment is not only photometrically accurate but also geometrically um represents and it geometrically represents the true scene that we see um next we have um Downstream metric right so one of the use cases we have is resimulation and so to go beyond just image based metrics we actually want to measure how low the domain Gap is in the

1:38:35 use case that it's going to be used for so one of the big metrics we use to actually Benchmark our environment creation is what we like to call reproducibility um it's pretty um intuitive in that what we are trying to measure here is we take the same driving model that drove in a scenario on the road we run it offline and we try and we measure how different the The observed um trajectory is offline to online and this gives us a measure of the do the perceived domain Gap by the driving model um between um the real world and the simulation um this while this does Benchmark the environment creation itself it also provides a um a metric on how well we're able to actually res simulate the robot um things like the latency um the ACT um yeah the latency the vehicle Dynamics model so it's not isolating just the environment itself but it has it's sort of a better measure of the full simulation system as a whole

1:39:44 um but nonetheless super valuable to actually measure how the Fidelity of the simulation um and another metric um is noral view synthesis so this is sort of a in some sense a privileged metric in a lot of in a lot of ways um because you want to somehow measure what the image quality is when you're away from the sort of training views you use to train um the environment model so um especially in in domains like driving this starts to become a lot very relevant because most of the captures have these linear types of

1:40:52 motion e e e all there at least for autonomous driving where we're able to effectively Benchmark um how well we're doing in these off axis Offa poses um and yeah we need we need to have this such that we can make progress in the field um so yeah I want to take this opportunity to then sort of go over I mean why we need this and what's out there today and how we um at wave are trying to um help make progress on this and solve this problem so if you look at um you know benchmarks in machine learning over time uh we see a pretty clear Trend here that once a benchmark is sort of established we get closer to human level performance eventually surpass human level performance and then we find that you know these data sets are not no longer uh effective ways for us to actually continue to make make progress um in the field so we can see this on the left we started with image classification we cross human level performance there was

1:44:02 a need for harder task to measure um how well we're doing so we moved the reading language and then eventually we go to competition math and once these benchmarks are established the community is very good about um optimizing for these metrics and eventually surpassing human level performance so if we look at image uh image classification I mean the classic example of how the um of how this started is image net and image net really arguably started or was the base data set on top of which the Deep learning Revolution started um we had lots of Innovations here like Alex net Dropout resets many many more um and it was really crucial in the early days of computer vision for us to measure progress in the field but eventually um we got very good at imag net and there was a need for something better so what do we do we sort of moved to harder perception tasks we moved from Global Image level classification the task like

1:45:03 semantic segmentation which required us to not only give a class for a full image but a per pixel class ID um for um images and we also move to tasks such as visual question and vision question answer visual question answering where it required a higher level of reasoning where um for example example in this pizza you have to uh actually reason about what's in the image and the contents of it to be able to effectively answer these questions um so once we moved to these tasks we were getting closer and closer to actual real world use cases um and these were very effective data sets and metrics for us to actually continue to measure progress um and I mean now the computer vision benchmarks are much more aligned with um how we actually are going to use them in the real world so if we if we go back to sort of noble view synthesis and we look at how the same Evolution happened here um I'm sure if you've trained a Nerf or a

1:46:06 gausian Splat in your life you've seen this tractor um it was a a perfect first example of how we can measure um and actually prove that this problem is solvable in uh in a small toy remain um and while it was great initially we measuring progress um you argue that I mean no longer sort of The Benchmark um that people are using to uh Benchmark New World um then we move away from these uh uh synthetic data examples to real world captures still very um controlled object Centric scenes um but moving in this direction of getting the harder and harder tasks such that we can continue to push the boundaries of what's possible um after that we started see even these examples we're um most uh methods out there today in the nvs domain are are pretty good at those problems so we wanted to make the problem even harder and start making things move so here uh we see some example of the early days of dynamic um

1:47:20 Dynamic no data sets um where people sort of took these uh synthetic objects and started um making some movements in them and try to see if we can actually get an algorithm that Faithfully reconstructs these um 4D scenes um and this is a a work that I really like um from um the folks at Bly um where they talk a little bit about how we can actually um quantify how causal and dynamic a data set is so if you look at the data sets I or most of the data sets I showed you in the last slide they fall in this right hand side category where we either have a very controlled setup where we have multiple cameras looking at the scene from different viewpoints um or we have teleporting cameras where we just have like random instantiations of various cameras in the scene um and and uh yeah but so these captures are actually they don't reflect how data captures are going to happen in the real world in the real world you're going to

1:48:30 have an iPhone you're going to move through a room or you're going to have a vehic robotics uh a robot that's basically moving through an environment and they're going to be capturing these videos in a causal manner um through the scene rather than having a teleporting camera or um having multiple Studio like setups um so we continue to see okay now we have we've seen methods that are able to um solve this problem in this control setup but now we actually want to move towards this left hand side where we have models that actually work um with data distributions we see in the real world um oh oops so one of the sort of big sources of causal data captures in the real world is autonomous driving data sets so I mean great works I mean that we've seen I mean kitty 360 new scenes weo weo open data set they're all uh great data sets uh for various purposes um and we see that the and uh the norb synthesis Community has sort of

1:49:42 you gone through these data sets to sort of further challenge themselves with data that reflects actual real captures and getting them to work in these environments we look at these autonomous driving data sets they're um causal their Dynamic they're outdoor so they sort of check all the boxes in terms of the difficulties of getting these methods to work in the real world um and so this is how the community is really starting to challenge themselves um but if we look at sort of how these data sets have been designed um they're mostly designed speciic specifically for the use cases of perception and planning so what this means is the sort of setup of the data set um the a distribution of scenes that are present are all sort of optimized for these task of perception and planning U and are not necessarily choosing things that are hard in the noral view synthesis domain um we also see I mean from the

1:50:52 previous slide that these uh rig setups are all um they're 360° rig setups but the amount of overlap between the cameras is quite minimal in the image on the bottom you can see the overlap between the cameras in the way more open data set um and this makes it impossible to actually have this uh metric of off access reconstruction quality because you don't have any redundancy in views where you can draw out a particular camera view and use that for this for the purpose of off axis metrics um and each of these data sets sort of have their own format and uh sort of uh storage and uh you know way to sort of access the data um and this makes a lot of sense given these are more general purpose data sets and have a lot of different challenges um but when you actually start to try to utilize these for uh nvs domains they start to it becomes quite tricky to integrate these data sets into existing

1:51:59 um Frameworks so um because of this we saw the need for a um new data set that's particularly tailored for the challenge of noal beew synthesis um that we call wave scenes 101 uh so wave scen 101 is an autonomous driving data set uh yeah specifically for this task of norval VI synthesis and I'll talk a little bit about the properties that make this data set unique um so firstly the camera setup so we really Tred to curate this camera setup such that it would be the most useful setup um for this task so as you can see in this image uh our rig consists of five cameras um four of them wide angle fisheye cameras the blue crosss you can see um in the image on the left um and in the middle of the we have this uh what we're trying to get to as an evaluation camera that you can use to actually Benchmark how well your reconstruction stay off the sort of training axes so if we actually look at what this

1:53:13 looks like if you look at the RGB views um of from a snippet of this data set you can see that this image in red is the sort front forward camera that's uh we use for evaluation and you can see on the two adjacent cameras on the left and right of the vehicle that's um can be used for trage so what this means is you have um enough overlap with the training views that you can actually effectively use this as a metric for off-axis reconstructions um but also makes the problem trable where you're not using like a camera that has no overlap and makes the task it's at the right balance between difficulty and um yeah difficulty and usability so if we actually start to look at the sort of uh the diversity of the data set and how we've tried to curate it such that it's suitable for this challenge of norb synthesis um we collected data across two countries so the US and UK um they and night um we

1:54:24 also tried to put a special emphasis on things like changing Illumination in the scene so you'll see scenes that a lot of scenes that have things like changing traffic lights uh brake lights coming on and off um blinking indicators and this is really done to reflect um the challenge of being able to Faithfully reconstruct this in simulation um Dynamic objects so you'll see vehicles but also pedestrians cyclists um and deformable objects that um are generally very difficult to um make work with scene uh scene graph type approaches where you sort of try to comp compositionally um arrange the scene um also things like exposure changes and variable distances traveled um in each segment makes the data set a good sort of way for us for the community to measure what we're currently good at and what currently needs Improvement um so just some visuals to um get give you a feel for what's actually in there so here we can sort of

1:55:32 see we also want to Benchmark static scenes so here are some examples of static scenes you'll see in the data set across residential and Highway um both posing their own unique challenges one you travel much further um and one there's just a much more detail and the vegetation and houses on the road um Dynamic Vehicles so again you can see two very different scenes here one in central London with very sort of complex behaviors of the vehicle um that vehicles that we're interacting with at lower speeds and on the bottom you'll see you know more of a American Highway type scene where things are moving really fast but maybe a less complex motion profile um in addition to this changing camera exposure um examples like going through tunnels uh lens glare these are all challenges that we have to um we have we face in the real in the real world and so we wanted to sort of reflect these challenges and Pick scenes that

1:56:39 specifically highlight um maybe more extreme cases of this um in the data set um deformable objects or as we like to call humans um these are um some examples of scenes where you know there's a lot of humans in the scene cyclist um and really sort of can highlight maybe strengths and weaknesses of a particular approach um yeah changing illumination talked about changing traffic lights um here we can see two scenes with temporary traffic light in the UK um sort of more classical American traffic lights um in the below and we've selected segments particularly where these State transitions happen and so you can actually start The Benchmark if you're able to model these State transitions effectively um yeah this is another one uh a tough one is night scenes and how we can handle those um when we do normal view uh synthesis um you can see some examples of this lots of challenges here um just lower light in general makes the

1:57:56 problem harder but also you start to see a lot more sources of active Lighting in the scene um such as the ego vehicle um headlights also things like other vehicles uh Lamp Post in the scene um and so we think this can also be effective to Benchmark works that are specifically targeted for things like low light or night settings um yeah so that's a little bit about the diversity of the data set you also want to talk a little bit about you know the structure for motion format uh we know in noal view synthesis Nerf Gan splatting call map is King uh sort of everyone most of the academic Community utilizes call map as a source of um posed information as well as 3D um 3D Point information so we've deliberately chosen to provide um all of our sort of pose and calibration and 3D um environment information in this call map format um such that it can as EAS uh be integrated into existing nvs code

1:59:01 bases and projects with minimal effort um here you can see an example um Nerf Studio a very popular framework for training Gan flats and Nerfs um you can see sort of the ease um and how how easy it is to actually start training um a model on a particular scene in the data set um this is all possible in about two lines of code where you just utilize some of their inbuilt functionality um to actually convert the call map information into a format that they have and then you start pring so this we believe will really reduce the barrier to be able to actually get more sort of people from the nvs community involved in actually benchmarking in autonomous domains and actually starting to um test how robust their algorithms have um so I mean now if we look at this table again um so we've released 101 scenes to begin with um we are look thinking looking to add more scenes in the future but we wanted to start

2:00:13 something with something small such that we can iterate and get feedback from the community on what's valuable um Within These scenes um we do now have an heldout camera for evaluation so we can start to actually get quantitative metrics on top of the qualitative evaluations we do for noval view synthesis um and it's specifically Built For This noal view synthesis use case um and so it makes integration with existing libraries much easier cool so just to wrap up um some final remarks um some key takeaways um so we've spoken a little bit about how learn simulators are key um to accelerate the development of autonomous vehicles um and why they're sort of needed to actually do scalable Clos Loop evaluation um for development um we've talked a little bit about how benchmarking of current noral be synthesis methods um it's primarily image based metrics and application based metrics but are lacking um noral view synthesis

2:01:23 metrics where most of the sort of evaluations are qualitative um and lastly we introduced um a new data set that we've released to tackle these challenges with benchmarking and noral view synthesis and we hope to inspire more progress and field with this um on top of sort of what we're doing at wave to uh to accelerate this field um yeah and this data set specifically curated for this task um and that's it for me today uh thank you very much um if this kind of work sounds interesting to you we are hiring so come get in touch with us at our booth um the team will be there tomorrow and love to catch up with you guys than I think we have time for one question fastest fastest time in the room cool um so if you could go back to one slide which was on novel sentences slide when you introduced that there was a scene where you had a between a graphic intersection yeah is it this one uh no I think it was

2:02:35 one of the earlier slides okay yeah like really at the beginning of the presentation yeah on the right hand side so one of the things that I noticed was there's a small artifact that appears right above one of the lights um on in the right hand image and my question is do you have pixel level control over the scenes that are reconstructed my my my motivation for that question is how resilient are the metrics that you're using to ensure that they a b accidental artifact that generated during the reconing process I see so you so uh if I understand the question correctly so you're asking how much controlability we have over the Recon like over over over the Reconstruction to to correct if there's artifacts if there's ways to measure accidental artifacts that appear in that process that way you don't include that in the training or the evaluation process later on yeah it's a good question so just to repeat it for online

2:03:36 um folks um the question is how we can detect or um yeah detect artifacts that occur um in the Reconstruction such that we can maybe exclude that data for use Downstream use cases um I mean one of the uh like we spoke about like uh generally measuring the quality we sort of use a multipronged approach here where we try to measure I mean if the artifacts in the sense of Downstream reproducibility use cases so that's one measure of how we can actually measure this domain Gap but I mean the sort of challenge is really when you're getting off AIS right so this is where we find that having held out cameras that are not along the view used is much more reliable in actually exposing these bad geometric reconstructions or artifacts in the reconstructions and that's really why we've tried to sort of go down this way of evaluating thanks right let's thank nikil and we're gonna have to move on but there is another break later if you

2:04:44 have questions just grab n thank you okay almost in time hello everyone I'm jca and I introduce myself today but that was gone but yeah I'm just look I'm one of the scientists at wave and for for majority of the time I spent a wave been thinking about word models and today what I want to do is kind of go through the Journey I've been and when I started learning about word models and to where we are today um so there's been quite a lot of talks in from Jamie L earlier today about how do we use World model what kind of world model will build so what I try to do here is this is ultimately a tutorial so try to cover a little bit more about uh the literature about word model so if you're interested in word model you don't know what they are that's probably a good starting point uh if you're an expert about world model maybe some of your working in here please give me a shout uh and if I missed out on some work still give me a

2:06:40 shout out I'll buy you beer sorry about that um so quick agenda again let's start with a I'll start with a brief motivationally why I think more models are important and then go through uh some of the what I would call Standard techniques or techniques that have been widely applied uh to World modeling problems and then dive deeper into like some specific work that I think it's particularly interesting uh in word models for autonomous driving specifically so again as I said disclaimer the literature is huge this is about 30 minute talk so I couldn't cover it all and probably I missed out on something and if you have great points for me I'll be very happy to learn more so motivation why why do I think world models are a good idea and again I always start with this picture you just look at this picture can you tell what's happening like do you have guesses what's happening what is the car on the left

2:07:40 doing is it moving is it stopped we don't know but we have guesses right then this is actually what happened is that car was stopped we went around it but there were multiple options there and was very clear to everyone in the room what these options were and that's because everyone has an internal War model we can access to that and we make decisions based off what we think is going to happen next so but where where is where is this knowledge coming from I think the interesting thing is like since the early humans start to like observe the environment then they learn how to interact with the environment and they basically learn these samples and they build this World model and this is not just this world model is built on any interaction and observation of the world and we use it generalized in so many uh different situations like many of us probably haven't crashed the car into a wall but we all know it's

2:08:37 it's not great right and we avoid that when we try so similarly machine learning models can learn similar Concept in a similar way which is through observation interaction and you know mostly observation interaction with their own environment or even observing other agents interact with the same environment or a different environment ET Etc and this can be done by serving videos interacting with the simulator observing how models interact with the real environment and so on so I think my motivation is like War models are part of the human intelligence and they should be part of of any embodied intelligence so I think that's why it's important to go through World modeling and understand where is this coming from where how it started how it's going and uh hopefully some of you have the answer of how this is going to go in the future um so let's start with a brief introduction of world modeling

2:09:40 techniques so I would say this world models this is paper 2018 I call it how it started in bracket I say with neuronet World model started way before 20 but I think this is the one of the first papers I could find that approach word modeling using neuronet um how does this work so it starts with if you look at the image on the top left so Auto enced right so high dimensional observations of the word get compressed into a latent space and decoded back uh through reconstruction L now we have a way of compressing high dimension High dimensional observation of the word into a very compact latent space through in this case was variational Auto encoder and what they did next here is to use the recurr Nets because I would say in 2018 that was kind of standard way of modeling sequences and temporal and especially like temporal sequences and these paper started proving that you can learn both a word model and a policy

2:10:48 through uh that scheme at the bottom by interacting with with the environment and you can achieve state-of-the-art performance so at the time that was done using the open iy environments butly proed that making this forcing a model to reason about the environment and what happen next really helped learning a a stronger policy uh I think immediately after that um dreamer B1 came out that was 2019 and Dre we want introduce this rssm architecture which is effectively a recurr net uh with some fancy distributions being matched I'm not going to too much into the details here I put links to all the papers if you want to uh read about all the details but I think what's uh really interested in dreamer it was the idea that you can learn a word model and a reward model which is observing uh the environment or even other agents interacting with the environment and then you can train your own policy fully in imagination

2:11:51 so then when you start training your agent there is no interaction with the environment um and they prove that by doing this what you can do is learn a very strong policy um and again at the time they use continuous Laten so was similar like a VA type of encoder of encoder to represent this High dimensional observations and they Pro using the Deep Mind Control Suite that you can achieve very very good results without even interacting with that environment um which is pretty important because uh when we think about embodying AI into a real robot that interacts in the real world we can't just throw a robot out in the wild and allow it to make too many mistakes before it's a disaster right and so this is this was very inspirational at the time um dream one was followed by dreamer 2 uh I would say the main difference between the two models was they moved from continuous states to discrete States

2:12:52 um and it was similar so they learn behaviors full in imagination from a word model that I could train from observations and this time this model achieve state-ofthe-art performance on the atar so right so we see that we move from one type of lat to another and there is extended capability and the dreamer s continues and there is dreamer B3 where in dreamer B3 uh they the authors start to propose a lot of tricks on on how to train and stabilize the training of this uh recurrent net which is uh not very easy to train uh I don't know if you try to fit some dreamer models they they're tricky if you succeeded it they do very well and dreamer V3 they they show how by stabilizing the train and allowing to train and Sail Out These dreamer models they can start tackle much more difficult environments like Minecraft so we started from very simple environment the mind control switch went to Atari and the end minecra so we kind

2:13:57 of see this uh Improvement in type of tasks that this the word models can help to solve um so I'll pause here because I've mentioned a couple of times discrete Laten continuous Laten etc etc so like if you don't know what these things are it's just a coding images and videos so there is some literature on how you can do that so and what's the goal here the goal is to take a large representation for state of the world like an image or a video and compress it why we want to compress it because then the world model can treat this compressed representation reason in this compressed representation much better than it can do in the high dimensional representation so I just put some pointers here uh but uh I'm expecting everyone is should be fairly familiar with this thing so we talked about recur n and we're moving now to Transformers so faki came out in 2022 and me similar to happen in language modeling where were modeling

2:15:06 sequences everyone moved from recur Nets to Transformers and we kind of started to see the same shift happening in the world modeling World um so yeah faki came out and it's like one of the first uh pap that prove that you can cast the word modeling problem which is video modeling in the end um as an next token prediction or as a Mas token prediction and here this like an example of what faki could generate this is like rather small images uh uh they're quite basic uh but also what they did is they started to introduce this concept that these word models can be prompted with text so the idea is you have a encoder that allows you to discretize images and videos now you have a sequence of discrete tokens and you can train your model to predict the next token or predict a mass sequence of this token in this case it was a mass sequence of the your tokens how it's time to generate you can just prompt your model to

2:16:06 generate uh what you want and I think this was kind of the first work that demonstrated that that's possible to do Transformers video prompting um SL agression Iris um so this paper introduced uh again similar ideas from the past which are from uh dreamer again prove that Transformers can be sample efficient for model so they show that have higher sample efficiency than the cor corresponding like recurrent net models and very similar setting of dreamer they manage to like learn a policy completely in imagination which you it's very safe when you think about real world deployment and they Pro that on the tar Suite they could do very very well so I think for now we kind of jump back and forth from model that want to generate videos model that want to uh learn a simulator of the world to to to train a policy uh so what I'm trying to hint is World models can be used for several things and you'll see like going

2:17:14 forward that that's kind of the case but yeah so don't don't don't be confused by the fact that some World models are used to learn policies and simulation some orere not so they've been they've been widely employed in the last few years and I'm trying to cover um the most interesting work that I found so far uh and of course I mean hasn't been claimed as a world model but this year we also Sora coming out as kind of impressive video generator and with sort we started seeing the shift from Transformer baser architecture still Transformer based but start to train these models using latent diffusion instead of next Tok and prediction and yeah sort of I mean we we also it like realistic long video generation from tax proms U I would say this been done on a known data sources uh hasn't been disclosed but it's probably large scale and so we see this shift is went it went from recurrent networks to Transformer to more

2:18:17 diffusion based techniques now um and last I would like to mention I know that uh the title of the talk is we're going to explore generative World models but I would like also to mention that in the world modeling literature there is a an entire stream of non-generative methods what does that mean uh for example this VJ uh work from 2023 proposes a different idea where you cast the word modeling problem not as a I'm trying to predict what's next which kind of takes away your ability to then generate samples but you can still learn a representation that is word model live in a in a non-generative way so I'll just mention it because it exists it's not really the the main topic of my talk but I thought it was important to mention it so this was kind of a very brief overview of how the world modeling framewor shifted in the past like four or five years uh and what happened is world mod started to

2:19:24 be widespread in autonomous driving so here I just put couple of images from a couple of review papers that came out recently that discuss about how World models are being used in autonomous driving um and actually yes everything here if you look at these images everything seems to start about 2022 uh well there's not much being reported earlier but I would say that before 2022 way we were like already investing in thinking about world models I would say very early on about like 2018 a way we published the blog where we managed to train a policy and deployed in a small car E I don't know if familiar with it learning the policy full in imagination on a world model that would be the real world era um and then we started investing more and more where we started publishing papers about predicting the future starting to trend is World models first starting from a simple representation like predicting

2:20:24 the future in a semantic segmentation space uh moving to like fiery that did it like in bird I view space I know and some of this work then started to get and we try to like kind of close the loop with what I presented in the literature before where in 2022 where we published mile so mile is uh recurrent State model very very similar to Dreamer that U predicts Bird's I view segmentation and action so this is effectively a model that has learned jointly a word model and a policy uh we deployed this in Carla and uh at the time I think we got state-ofthe-art uh results of course things are going much much better now and similarly at the same time there was this work called Sam 2 um which is very very similar to to my I just wanted to mention it because I think the two words came out basically one week apart we didn't know they didn't know we just found it and we just found it so that's another another

2:21:31 example where uh using these recurrent State models in Carla like people were able to fit word models and a policy join um right so then we moved on from that and the main reason is we Tred to scale up to real world applications like and we didn't really see the scaling curve that we wanted so we decided to reink what we were doing and that's how we started working on Gaia so Gaia is our way of rethinking scaling word models with data and compute and the what's gu guy is a word model based on an auto regressive Transformer and gu emplo a lot of encod different modality encoders to compress images actions and and text as a set of tokens H once you do that you can just take this multimodel sequence align it temporally and just model this as a secet so we did this last year we presented it at I think vbr last year and I think is in most of our talks uh today we are now using it and conjunction with prism like for uh

2:22:48 several things a away uh yeah what was impressive when we first saw is like we started seeing that this new world model that we were training started to have a lot of nice things that we wanted to see in World model they started to show us that they are intelligent they do understand the world and this is one of the examples right they started to demonstrate us that they do understand multiple options um and so you can take a word model you can initialize it with sequence and start unrolling and if you take a few samples you'll see that there is quite a lot of diver uh and you know this example I think it was in J St as well but it's one of my favorites because here is just the word model it's demonstrating as there you can understand even interactions with other agents can be different there are like multiple options out there and having all this information is really really important when it comes to like make decisions for

2:23:42 driving um and here I mean you can see that these video Generations are not like Pixel Perfect but the concepts and things that are really really important that we should care about are there and that that was really remarkable when we first saw it we were so excited um again I think this has been shown today again but another mind blow mindblowing moment for the team it was when we first saw this this result we could force a ego vehicle to drift in the in the wrong lane and gu could simulate other vehicles responding to our actions um again all of this is fully un to end nothing is and it is an emergent behavior that you get from training at stale um again we allowed Gia to generate from tax prompts and you know here is some example where we just say I want I want the weather to be sunny rainy fogy there SN way and we can get very very realistic samples out of it uh similarly can for the time of day

2:24:46 and all the sample look realistic uh and respect the prompts I would say one other important thing here is like the long generation was 2x uh is a 2X speed up in this video but here we just start we just started generating a video so we kept going uh this goes for like minutes in the future and it was one of the first time that by looking into the world modeling literature that to find something that is that stable for that long in terms of generation so that was about like a year ago um so after that we started seeing a lot more work coming out in uh word modeling for autonomous driving uh this is just one of these examples and we started seeing that there's been a little bit of a shift uh towards like Laten future so Drive dreamer is one of these examples so Drive dreamer is like Li diffusion model that allows to generate videos from HD Maps 3D boxes or actions um so and it can also predict future

2:25:57 actions so once you can do that you can look back your next action into your word model and enroll from there and this has been trained on like an evaluated on new scenes and here are just some examples of what they can generate uh condition on like a 3D scene um and yeah it's truly remarkable uh to see that so many different techniques can be just strained with data to understand the work that's it's just kind of a proof that the intelligence and all the concepts from the world that you want to learn can be learned just from data um here's another example where uh they show how once you put back the action that you predicted into the word model you can drive within your own imagination uh so what is show the bottom is the velocity and the steering angle predicted by the model and at the top is the video predicted by their own work so they can basically learn to drive in imagination after

2:27:01 that um similarly here like Drive World model another latent diffusion model uh the main contribution here and difference like we started like all the videos I show up to now they like single camera right when we think about deploying uh autonomous driving to the real world all these deployments do have multiple cameras see seven eight nine multiple sensors and they're very high dimensional so uh dve War model I think is one of the first model that came out I started demonstrating that this can be done uh with multiple cameras the geometry across the old cameras is fairly consistent with no prior geometry into the model uh and also this model could also generate actions and was trained on nin and the caveat here is you can see that these videos are a little bit uh jittery and that's because once you start adding cameras you can't U simulate at a very high frequency it's very expensive and so

2:28:01 these mod runs at a frequency that is lower than of the MoDOT size short before but you know they extended it to six cameras which it's really impressive and again if you look at these images and pay attention to like the consistency it's quite it's quite remarkable

2:28:26 um right and then a a driver I came out so interesting here we started we started to see multimodal language models being added on top of word models I think this is one of the first I've seen uh and was TR on Newen and some private data sets that have been disclosed but uh the video generation here is condition on actions and these actions are described by text and there is a large language model involved into the generation I thought it was quite a novel idea we never seen that before uh but we started to see like diffusion and multimodel language models coming together in the world modeling for um and moving on from Real World images uh there's been quite a lot of work as well trying to do word modeling in a different space which is this case for o word is 3D occupancy and here an interesting thing is like what we started to see is like moving from like plane Transformers similar to like bits

2:29:33 or uh stand Transformer to like spal temporal Transformers right because when we think about the word and the word model both space and time are important while up to until uh SD Transformers most of these work at model space and time as a unique thing and that's that that's more expensive um so award started to do this uh using 3D occupancy Maps um I think after that W published for pilot 4D uh and still this is the diffusion uh model uh the observation space is like lighter space um think it was yeah train on new K and ARG so like more data sets lighter Point clouds like a different like observation space but again all all all the same ideas of trying to model the world and in a certain representation space and try to predict what happens next and here's like great example from examples from their blog where they enroll time and they show how they can predict um lighter Point clouds in different

2:30:44 scenarios uh and this this prediction is quite accurate and like for a few seconds in the future which is probably what you need for like plan um finally I think this was mentioned by uh someone else today like very recent paper I think came out like couple of weeks ago uh and was started to see like more diffusion models coming into the world of word modeling and started to tackle high resolution video generation so this model Vista it's quite impressive video generation at a very high resolution so this is the highest resolution I've found in the literature and these are some example videos from their work quite stable generation and finally also this one came out we that um I think the most interesting thing about this paper if we're starting yes they do propose a new generative model to generate uh vide uh but the most interesting I think uh the most interesting part here is they just don't limit themselves at

2:31:55 showing that they have new world model that can generate realistic samples they try to close the loop and closing the loop means they show results where if you analyze your failure patterns on your endtoend policy and you have a way of generating new data samples that tackle those failure cases you can actually improve the performance of an end interesting result here was they demonstrate that you need both you need a really good word generative model and a really good strategy to identify and generate the samples that are which you are with and right have a look at the paper but yeah the results are pretty clear you can just take a great word model generative model and Generate random samples throw them back into your training set that's actually the grading your performance uh and at the same time is you can generate samples of things you fail at from a know generative model that is not to once you have both you

2:32:56 start to see the Improvement so I am almost at time so yeah let me just wrap up what I try to discuss today yeah I try to stay away from like just wave specific things I wanted to like kind of try to cover a little bit what's out there because you know World models are starting to be out there especially like in autonomous driving we're starting to see more and more companies or research La started to investigate World models and invest a lot of them so what do we have seen today here I would say yeah we should probably I hope everyone agrees that now World models play right now an important role in embodi and will play a massive role so it is important to invest in them and we have seen I would say four main techniques uh that are commonly applied in the world model like something like dreamer and recurrent models we started to see like vector quantize representation that can take large uh spaces like images or videos

2:33:59 and compress them into like token spaces started to see the fusion models and we also briefly seen japa models which are the non generative models to learn models of the world and we've seen that all these techniques can get like put into the into work for autonomous driving and they can employ the data generators they can employ to understand the word as representation learning tasks they can be coupled with behavioral learning they can be used for occupancy prediction and further planning so applications of world models in autonomous driving are widespread and so yeah I would just like to leave everyone with just one question which is how do we improve them how do we leverage them to unlock the full potential and body ey and that's just an open question the room thank you very much and I'm happy to thank you J for a great talk you mentioned consist frame frame and then you also show I think

2:35:14 maybe one of the GU videos and you said we let it go for a minute you have an idea how model I the question for online it's two questions right so one is if we have a sense when we generate from gu minutes whether we look back to the the same place um and the second question is for other models like lingo does lingo understand we went back to the same place and can tell that that so uh to answer the first question is like no we never seen really like looping back to the same exactly the same place and the reason is um Gaia is effectively a a Transformer modeling just sample the next token so in order to go back to exactly the same place the way we model one single time that is like thousands of to tokens so unless you sample exactly the same thousands of tokens you not same place the combinatorial space is exponential and so usually we never seen guy going back to the same place uh we have seen early on uh when we were

2:36:35 experimenting with it uh guia trying not to be very inventive and mostly like try to remember some places of the world that we have seen more than others uh but we immediately started seeing that when we start scaning this up that goes away so stops memorizing and the reason is at least my my understanding of this is when you expose these models to significant amount of data they can't memorize them anymore right so and neuron networks are extremely lazy objects so the easiest way of solving that is not memorizing what you've seen but is really synthetizing and like condensing the rules of the world that generated the data and now they becom amazing data generators and so we stop in this memorization and going back to the same place kind of behavior uh regarding lingo I have to be honest I I don't know I don't know if we have an example of going back to the same place and lingo saying we just go back to the

2:37:33 same place but in general like all these models have fairly limited memory so even going around the same like one block and back probably takes minutes so it's very unlikely that these models until they start fully generalizing to extremely long context we be able to do

2:38:08 so uh the question is we try to see well generaliz by trying to drive in reverse and yeah the answer is yes yes we try to like Drive In Reverse and because it hasn't been train on a lot of data where we reverse what it does well we have two samples interesting in one sample we started reversing and G did the U-turn because that's how you start the U-turn and in the other one it stopped reversing and it started to go forward uh but yeah I think we we never show it publicly but like we kind of stressed that guy a lot more uh to the point that at the time of publishing the paper what we did is we took one image of everyone that worked on this Projects Home time town a road in their home town and we started rolling from there um so what we saw is it was mostly train on UK data so when we initialize it on right hand driving what guy usually did is R to the left and kept driving uh but yeah he

2:39:05 generaliz it it generalized quite well of course the image generation was the same quality of the data we trained on but like the concepts and even the behavior of I initialized this model in on the wrong side of the road according to this model and this model decided to go back to the right I was kind of impressive we didn't expect that started seeing all all of these emerging behaviors um during the stamping of the future in the Gia model we are shoting from multiple possible Futures uh this include the behavior of other agent like uh one if one vehicle is cutting you or not or you you uh I was wondering if we have more deterministic control over the behavior other agents during the sampling of our um so the question is um when when we uh sample future from guia and the future is multimodal we do not control other agents behavior and so the question is is there a way of controlling having more contol yeah

2:40:06 actually yes there is I think uh I haven't included in this presentation because I tried to limit uh my tutorial presentation to paper that we published but I think today jimie gave us ni people how we can do that uh with G which is for example the examples that we gave on prism like we can add agents there and control what they do in a video um and so yes we do have new versions of GA when we're working on full controlability of other agents through their 3D position text etc etc so it's it's something that we have in the works but uh we're adding the ability to control better the behavior of other agents back what your thoughts on training a solar based on theate representation inside the um question is for thoughts on trading controll or driving policy Imagining the Laten space and yeah that's that that definitely like interesting thought I mean a lot of these papers that we've seen uh have

2:41:08 done that so I I don't think it's un like not plausible um one last question uh you mentioned that the rssm and M scale on you mention what about scal efficiency of it right the question is um I mentioned that the when we try to work on rssm we didn't see uh scaling with data and comput and what's what's your reason for that so uh I believe that the main reason is like rssm used to be trained with this idea of you have a prior distribution or what's going to happen next and a posterior distribution so and then you will match those with a K diverence so when you looked at the all optimization landscape you like your reconstruction loss two moving distributions and a KL loss which if you don't tune correctly collapses so what we've seen is it wasn't as easy to just say okay let's double the parameters let's double the data uh you start training everything seems fine but if you end up in that weird space of your

2:42:14 optimization where that uh those two distributions collapse on top of each other the prior and the posterior you're doomed you like you forget everything and it was we seen that with scaling uh we couldn't really control that easily so it was really hard to tune all those hyper parameters uh whilst when we switch to Gaia we reframe the whole problem as that's just next token prediction and so there's only one loss it's the cross entropy loss and you guarantee that lower is better right so in in in the in the rssm formulation you don't have that guarantee because what is better like lower KL better reconstruction you have to trade one off for the other so it made it very very hard to scale just because you start an experiment you start training well and after a couple of days it collapses on you and then like okay let's start from scratch change your ryer parameters and go back and forth while with Gaia what

2:43:07 we saw is that double the parameters scale a little bit The Learning rate uh go and you see that loss keeps going and if you don't stop it keeps going right so I think that that's what allow us to rescape all right let's thank again Juka for the amazing talk and uh we're now we're not having a coffee break for 15 minutes uh please come back at 4:50 for the there are two more talks on language for driving and also the future Direction ofp driving all right um our next presenter is is a principal of the stist W leading the legal team are yours all right they hear me well all good yes U okay uh yeah so my name is Al and I'm going to tell you about the language uh uh in autonomous driving uh so let's start with this uh kind of main question why the why would you use language in cell driving that was actually completely non obvious just few years ago right so uh it's all it all started uh at around 2018 actually

2:44:33 with a a Berkeley did Drive X data set it's the first explainability data set for cell driving and uh in that first paper uh regarding explainability this is a citation from that paper uh they uh side those main three reasons why would you need uh language in cell driving uh first as user acceptance so driving is a radical technology so everyone needs to trust it uh people need to understand and extrapolate the behavior H and also effective communication between a vehicle and a human uh uh then uh another thing is that uh before uh let's say 20 23 uh there was the main reasoning architectures before LMS were mainly those kind of Technologies you guys probably are all familiar Monte caret research there also a lot of planning uh in deep learning using graph neural networks uh and the mixing of semantic engines and uh deep neural networks uh but in 2020 uh there was this discovery uh where just next

2:45:45 language token prediction can makes model uh reason and this is uh gpt3 not 3.5 uh where already in 2020 it can do a very simple reasoning uh regarding cell driving right so you can see that I'm on the three Lane Road in the Middle Lane I need to turn left uh and what do you need to do next right and it autocompletes this plan and this plan kind of makes sense yeah there's a small mistake but it's kind of okay for 2020 uh but think about it uh it's completely nonobvious why would next token prediction make uh any model to reason and plan right so when I seen this I finally realized all right so then this this is a technology language model technology is the one that would allow models to do planning and reasoning in one single architecture without without any extra Machinery like tree search or gnm and so on another Insight is that the explanation is actually kind of equal equals to Reon reing right so

2:46:50 think about it imagine you see a person and then you press on a break and then a model generate explanation okay I'm pressing on a break because there's a person on the street right so if you just do this uh in C kind of simple thing where you place this explanation before you do the action suddenly the sentence become becomes a reasoning uh sentence where you say okay I see the person and I'm pressing the break because there is person on the street and then you press a break right so the single thing that happened is that you kind of uh swap the places here between explanation and action and obviously explanation now does don't have access to action it's kind of a bit tricky but you can see that explainable model actually is a reasoning model which is kind of non obvious fact that I became obvious when LMS kind of came uh on on the scene okay so uh yeah another thing is language is a very nice compressed

2:47:45 information medium so take a look at this very complicated scene in central London uh while it's all complicated uh it's actually quite uh it's not so uh long to just make a really nice description and text about all of that my Lane is straight only the light is on my Lane is green I'm just attending to pedestrians speeding it up if you just compute those words it will be like just one 140 bytes right which is a uh it's kind of uh check information compression but that's pretty much everything you need to drive in the scene and because it's such a nice information medium there was this uh whole field emerged about uh that is called LM agents that take language and use it in all sorts of different ways right and uh we in C driving also can use uh old Technologies from LM agents to endow our models with those many Advanced features that take advantage of of this medium you can have long uh term memory in

2:48:47 context learning lifelong learning uh you can interact with the user obviously and I'm going to show examples of all of that uh and the last but not least is that uh hk's data for sell driving is uh particularly precious uh you know to actually drive well you need a lot of combinations of different objects and a lot of weird things going on so people behaving weird some animals on the road and so on and uh uh before uh large pre-train models it was kind of nonobvious what to do so you just as a company or lab you have to collect all that data but now uh with the release of all that huge models that we can all use uh hopefully they still open source at the moment right there is like the second uh option for many robotic labs and Company so before you have to collect this data yourself but now you can rely on the large model that are pretrained on that data and you hope that you're just going to

2:49:52 generalize those driving cases okay so then uh uh how would you add the language model into autonomous driving system uh so for that uh tutorial I reviewed many papers and I I'll share the slides later so like there are a lot of links just just to show that there are a lot of work actually uh uh going on in that field and um uh I mainly uh picked some papers with which are about uh pre-trained language models and uh that appeared after 2023 but this is the prototypical system uh how would you do that uh so uh you take the input from kind of autonomous driving sensors could be multicam Vision or just a scene State lighter uh information you pass it through this language Bridge which uh translates this information to token then you use a pre-trained language model and you pass the output of that through the another action bridge and that uh makes a driving output that you can pass to the car Additionally you can

2:51:01 have a language output which obviously tells you explanation and reasoning and on top of that you can add a lot of agentic infrastructure from a agents field and I'll show you different examples but before that uh how can you measure so before implementing anything right you want to establish a nice metric so the first thing you to do is obviously how well it drives and there are several simulators like Cara new scenes there are new upcoming nerve simulators that nikil was talking about there's a highend uh environment which is simplified Simulator the second thing you want to do is to measure how well vision and language work together and for that there is a this beautiful picture on this drive LM uh GitHub and you can see uh where they show all sorts of data sets uh that test video to language conversion specifically it could be VQ or explainability here's a bdx data set there is also lingo QA data set that

2:52:03 we've released uh already uh and uh and one specific thing that we also released uh is is this new technique where uh to test how well uh model answers uh a certain question we okay we train yet another llm as a judge that just predicts correctness score uh because when we started to work on vlms we found that typically used metrics like meteor blue and so on are not really capturing all the semantics forell driving they're capturing the sentence structure but they often lose the nuanced meaning like okay was the car actually left or right like one word can matter a lot uh in in such in such scenarios and we found it a kind of fruitful approach where you just actually fine tune yet another llm to uh compare between U VK VM answer and human answer and then just gives you a score and we release the paper on that so you can read about this and it leads to very nice correlation with humans uh so another thing you want to

2:53:10 measure which is very non-trivial is the whole alignment between Vision language and action so if you think about like three modalities actually so it's not only uh Visual and language but also action comes in and it's all happening in a closed loop and it's very tricky uh What uh how to measure all those capabilities but there are new benchmarks coming up so there are at least three I listed here Lang out L pilot and Dorothy and some of them measure instruction following some of them measure reasoning and enclosed Loop um so you can check them out all right so uh just kind of a division of all the work that happened um in llm based driving uh the the the the initial the initial papers that appear they mainly focus on this agentic infrastructure where uh originally there was no really many strong uh VMS visual language models so people uh used CH GPT uh using API just trying to play with

2:54:18 this idea of U how to use language for driving uh and uh in in this picture I'm kind of labeling in green like amount of focus or amount of research that put in that was put into uh one of the aspects of the driving so you can see that in those papers uh majority of work is was put in a gentic kind of llm uh text based infrastructure so here are some examples this is the uh the earliest paper I was able to L actually uh it's called drive like a human and uh in that paper uh uh they use this high highway environment uh they take um observation from U different Lanes put it into text and let CH GPT reason about it right so CH GPT can generate different discrete actions in text and then you parse that and you put it back and even already there uh quite some time ago actually it was like in a summer 2023 uh they already did very Advanced uh kind of reasoning techniques with memory so you can see that okay so you

2:55:31 see those two cars and um first uh chpt ansers okay I want to just stop and uh uh wait for other vehicle to pass but in reality you can just go because there's another Gap uh they implemented self-reflection where uh there's a human kind of Correction for that reasoning and NG GPT reflects on that saying that okay well there was enough space that is added into the memories and then later that uh uh next time the you you see the you know the same kind of situation you use that memory self-reflection and so on uh to correct your decision and actually implement the correct driving decision so you can see that this is very sophisticated reasoning already happening with very early jgpt models just using API uh and a few other papers along those lines is so for example agent code driver uh here they added uh yet few more advanced features for example that the module so that you can communicate

2:56:36 with words with other vehicles which is again just takes advantage of the language being this very nice compressed medium you can just say to everyone else saying okay I'm just changing lanes to to the left and other uh cars being also autonomous ding by LS can understand this and react appropriately uh and you can see that there are quite many features uh uh this this paper introduced it was a memory and lifelong learning self-reflection veriable communication with others and Chain of Thought uh yet another Mo uh paper along those lines is called the language agent uh used another technique from llm Agents which is usage of tools specifically they had several tools available uh to them for example you can get a a leading object or some kind of predictions and then take decisions based on U uh based on all those tools uh use some kind of reasoning uh and even use a retrieval augmented uh generation where you uh use

2:57:47 some kind of embedding to uh to get the uh most common text uh descriptions of the scenarios that you have seen in the past um so next uh set of papers uh focused on kind of the right part of this diagram where instead of focusing on agents uh people focused okay how can actually can you transfer uh action how can you parse action from llm into driving output and this is is a non-trivial problem so first papers we're just trying to parse very simple uh discrete actions uh but here I just found few more advanced papers for example language MPC is the one that I quite like uh is a paper where llm actually controls parameters of NPC controller uh and can take this high level decision like tuning coefficients for NPC and specifically when you do this you get those Advanced features like a driving style adjustment for example you can say in a pre- prompt that okay I'm an aggressive driver or I'm a conservative driver and

2:58:57 um deciding uh LM would appropriately tune the uh parameters of MPC and uh do kind of appropriate behavior uh and another interesting two set of paper slang prop actually from from wave from uh our intern and LM pilot they went into another uh kind of approach where llm generates generates code that is then executed on a closed loop simulator so instead of generating simple decision you generate the whole procedures which is a very interesting idea all right so then uh after that uh several VM VMS appeared and um here's a small review of those papers uh uh this is the kind of the general architecture of VM that drives a car so here instead of uh text you actually have Vision that goes through vision token Bridge uh and uh and then later goes for uh Vision language model and some action and in those papers people did not really focus on this agentic infrastructure but mainly focus on Vision so this bridge is the main kind

3:00:18 of focus of the research in in this area uh so here is a few uh papers that I selected first is Drive gp4 uh and here we can see this typical scheme where uh you take in frames you use visual encoder and typically what's used this clip like everyone else uh you know people found in so driving that okay well the best visual encoder that transform images in text is clip uh then you take that you have a learnable projection projections uh which uh uh generate tokens for large language model which you can train or uh find tune with Laura or just freeze and just fine tune projections uh and using that they got state-ofthe-art on bdx uh so this is where we first have seen that u a large language models uh get state of dart on explainability data sets uh all right so the uh we also at wave had a a very similar project very similar idea which is called lingo lingo one uh so we released actually an

3:01:37 academic version of lingo one in the paper lingo K and uh architecture in that model is very similar also we take images pass it through clip pass all that through q forer and have large language model specifically the best was vuna uh with that paper we actually released the data set training data set we released uh lingo judge that I mentioned before uh and various uh just ablations of very different parameters that you can tune here and here's the demo of lingo one um this is a more kind of stated version but still can uh participate in the dialogue so you can read here uh so user asks what's the weather like and um lingo says it's rainy do you drive differently in Rain uh yes I need to be more cautious due to rainy conditions how come due to raining conditions the road might be slipper invisibility might be reduced so when will you go I'll go once the black sedan in front me start moving so you can see kind of see an

3:02:43 understanding here uh and then it kind of goes into this generic Rand right so will when will it stop raining it says I cannot predict so is it true that it always rains in London and it says no it's not true which is actually correct today was a sunay uh so the next uh section uh of papers are uh uh like a set of papers that focus on uh perception and um in that said uh people focused Main on how can you generate not only language but also additional uh perception output uh and typically it would be bounding boxes or segmentations so uh in in that approach those two areas where how can you put uh uh kind of vision into the language model such they can predict boundary box segmentation are the main areas of research and uh uh one of the most advanced one that I found actually is the uh this paper called important understanding of driving scenarios and uh uh the guys here uh trained on many data sets uh uh well

3:04:07 data set included uh new scenes way more UHD YouTube and so on and uh it's actually quite sophisticated architecture that involves this idea of time tokens such that a model can uh have temporal predictions not only responding to uh what it sees now but also predict in the future and uh uh refer what happens in the past and also uh generate some some perception so here are uh in opinion quite sophisticated examples uh of what it can do so for example you can ask what's the 3D spatial location at at a certain pixel 2 seconds ago and uh the model correctly predicts uh uh what happened uh at that location and it was a pedestrian that location so it identifies a pedestrian and says where it where he was uh it can do also regular action decision so you can just query by a bounding box uh what do you need to do uh for that specific car right and uh it's correspondently figures out that you know uh where this car is located

3:05:28 and what to do with it uh and it can do to tod2 3D kind of uh a transformation where uh it clearly shows that it understands 3D uh World it becomes really sophisticated perception system uh and another paper I wanted to highlight is called hilm D and uh the main uh challenge that they addressed is that in C driving as opposed to uh like many generic video uh understanding systems uh you really want to pay attention to small uh objects which are very important and those are traffic lights and traffic signs those are typically quite small they could be far away but you want to really you never you never want to ignore them and uh that that specific paper uh focused on a vision system that has two different branches so with a low resolution high resolution videos uh such that from really quite far away uh and like using just one pixel they can predict that there is a red traffic light uh you can see like a very small

3:06:38 bounding BS here uh and uh hence it's a very nice solution to to do this kind of detailed perception uh we also at wave uh played with that idea of uh referential segmentation and what we did we attached the segmentation model uh to lingo one and this is a small demo of this uh so you can see a user is asking what are you paying attention to and the model not only responds to that but also identifies two regions A and B as a sign M mation masks and says that I'm uh you know paying attention to driving driveable area and the traffic in front and then you can uh ask even more stuff and and said okay well now I'm paying attention uh so you can ask about objects in the world like what are the broken white lines mean and you you you've seen that it selected the lines correctly and predicted what it is all right uh but the most interesting models in my opinion are the models that actually can drive end to end in a

3:07:49 closed loop and use visual Vision language and for that just small digression is that uh if you think about why do we trust human drivers to drive a car if you think uh so humans are just they just pass a theory driving test that takes at most one hour in some countries and some countries even less and pass practice driving test with also at most can can last one hour and uh if if you think about it this is way way less than uh what we do in the self driving uh companies where we test every model orders and Orders of magnitude more than one hour and obviously right now VMS can just go and pass a driving test I've been shown few times on YouTube you can uh check out the gp4 passive driving test uh so why why do we trust humans so the one thing is that uh humans share all common knowledge with us right so like other drivers share knowledge with with us so we trust them for example we don't

3:08:54 allow kids to drive but now you can see VMS have that knowledge now so we we should kind of trust them more uh another reason that human humans can explain decisions uh but this is actually a bit misleading because if you have a VM that just explains decisions it's not obvious that those explanations actually relate to actual action that the model took so you just have like two robots speaking one is driving and other one just tries to explain what's happening but it's not clear that explanation is related to the action so the reason we trust human explanation is that we know that it's all in the same kind of brain and same subrate and it's all happening in a closed loop and this is I I believe the the main kind of the ultimate system where uh the system can predict explain and so on and drive together through the same weights through the same network and those are the VM based papers one of

3:09:50 the early ones is called drive anywhere where uh where researchers uh uh used the again very similar idea of of a VM and um uh well it's mainly clip based kind of model but it's actually was tested on a real car where the network actually predict control signals uh another uh very great paper uh along those lines that's called ml Drive I already mentioned that uh so here this is this prototypical system where you take in uh video again for the very similar clip and Q forer and so on but then also you generate uh tokens of actions and then uh what you can do is that you can uh test it in a closed loop which uh uh those guys did so uh yet another innovation of this uh paper is that they also took ligher input and put that into the llm uh uh they use specific pre-training kind of uh step first to align vision and ligher um to just get a nice visual features and then put it in the language model so you can see uh people started

3:11:08 to like test those models in Clos Loop generating actions through the same weight uh another interesting one is called ADH and here you can uh so here the original idea is that there are actually two models one is a high level VM that can uh do reasoning and has way more parameters so you can uh go answer questions but then what what happens is that model uh modulates another lowlevel controller uh which which is also a VM but it's way smaller so it can go and drive a car so specifically here it predicts way points so is like hierachical architecture but also can drive Clos Loop uh and we at wave uh recently released this Al lingo 2 uh model and this is the first model that first language model that ever drove on just real public streets right so uh it's been tested in the real world you can see the text generates is kind of expl explanation of its actions uh and in green you can see the

3:12:18 trajectory that it takes so it's a first VM that uh can uh explain itself but also take action through the same ways right and this is hopefully leads to better alignment and and explainability all right this is uh kind of very high level training recipe of lingo 2 so first we pre-rain a vision model then we align language and vision and later obviously with that add action through our own uh kind of uh driving processes and we get uh full model that can do language uh obviously vision and then driving and uh the last paper but not least just very fresh so uh recently our intern Kine uh Katherine released a uh model called car laava and uh uh we got a first first uh place on a car autonomous driving challenge with that and uh this is also based on language model specifically llama uh and this is a small uh demo of that so you can see this is also a model that can go and uh drive a car uh just goes through some tricky

3:13:39 situations and uh there are two flavors of that model one doesn't use language uh one does and this is the one uh where it generates language uh so you can see you can it auto regressively generates trajectory and then language token so you can see uh highlighted explanations here just scroll down a little bit um oops

3:14:21 yeah so you can see pedestrian it highlights The Pedestrian and then uh it correctly identifies construction sign uh sign with explanations below that is going to go and go around it so so you can see the Dem later all right so as a conclusion uh so first of all there are a lot of unsolved problems so this is all work in progress Very Fresh Field llms and uh autonomous driving there are measurements problems we know we don't quite know how to robustly measure Vision language action alignment there's not a lot of high quality Vision text action data it's mainly Auto labeled it's not human labeled yet I'm not aware of like any big human label data set so uh clip is not really great for S driving uh one of the lessons is that uh we would love to have um one of the lessons of our team is that would be great to have a model that is really good at spatial reasoning whereas clip is more like U sometimes X as a bag of

3:15:23 words right so we just want something spatial and can reason in language uh also large language model is not super efficient right so you don't want to run like 70 billion parameter model on the car so there's a lot of work needed for efficient uh inference uh and in general uh there's not yet like one big model that has a gentic uh uh features and can drive end to endend this is like a small table uh I made so in green it means that uh some feature is implemented and you can see there is no single column that is all green specifically features are closed loop evation can run a real car can uh you know participate in a dialogue referential perception agent capabilities so so far yet there is no single uh paper or work that can cover everything and be like this Holy Grail of llm uh visual llm in in cell driving that can do all of that together all right and just final uh thoughts like uh just a a

3:16:31 handwave argument kind of uh uh for uh LM and cell driving so before uh so there's always this question in the field like do you need AI to forel driving right and there were always two camps and uh you know there was like really no uh winner right some people say like okay uh you know you don't really need you know any uh human level intellect to drive it's uh it's too far away anyway we can't afford it and maybe uh self driving is like a narrow task right so it was called narrow AI right uh but people who tgu for AGI uh the good arguments are that RADS actually designs for human designed for humans it's better to have human intellect uh and uh well Another good argument is if you do solve AGI then driving it's like an easy problem uh and if you have if you think about like a lot of H cases that that um happens during driving you know that humans actually quite they use all the intelligence they have sometimes right

3:17:36 sometimes people drive reactively but sometimes you need logic and so on but after 2023 some of the arguments on a camp without a kind of went away right so first all AGI is becomes like closer and closer uh and we have a lot of released pre-train models so you don't have you don't have to solve it yourself right so hence uh I think it's a good idea to like jump on this uh train and and claim that Yeah we actually do need AI for of driving so let's try uh use this technology VMS which is uh kind of the closest have fil driving yeah thank you that's the end of my talk hello can you hear me now yeah mhm okay so I would like to know uh regarding the llm so as you know lm's hallucinate so when we use them for reasoning and based on that we are trying to utilize for uh like we are taking that as a promps for driving capabilities so what do you think about the uh safety implications of that if

3:18:50 they let say hallucinate about something that can actually make the driving maybe unsafe for humans so what do you think about that yeah thank you yeah yeah so uh as I said you know mixing llms and in autonomous driving is a extremely you know early field of research so I we kind of don't know the answers on on those questions everyone in the world try tries to reduce hallucinations so one kind one cool thing about using uh pre-trained llms is that you kind of piggy back on a lot of other researchers trying to solve problems for you right so instead of uh you know trying to solve it yourself you you know that okay a lot of people you know working to to make llm solution at less right and uh so another uh kind of view is that uh no matter what you want to to have really bus testing infrastructure for all your so driving models right so we and the companies I mean since we're doing endtoend driving we have to have very

3:19:54 robust testing infrastructure Offroad evaluation like uh um in uh neural simulators and onroad evaluation Z and uh uh no matter what whether we use reasoning or don't we have to have that done and uh hopefully that catches all the regressions possible right so we really investing into uh all sorts of evaluations but as I mentioned evaluations is not really yet a solved problem specifically for LM so there's not much work done on like reasoning for example hallucinations and still driving I'm not aware of any good Benchmark yet so yeah the single it's not a great answer but yeah it's a kind of work in progress yeah thanks for the talk so actually I have a question so so noway there is another like root ways just using the world model things right right so in terms of this well am strategies or compare with the word mod strategy what do we think which one yeah yeah so well I mean we are pursuing both of them

3:20:58 uh at the company and uh you know obviously Evena presented this paper where uh people started mixing those right where a single model can generate language and uh uh video at the same time so there are ideas of putting it together and uh try to take advantage of both models right so there are papers uh uh I forgot already the name of it claiming okay you can do reasoning in the visual space right the fact that you can predict the future and you can predict different Futures this is how monar research work even right works is that by predicting multiple future and figur out what's the best one it's also certain well it's a reasoning right it's planning and reasoning kind of paradigm so uh there's definitely a lot of ideas how to mix them together and I don't think they are uh you know they they're complimentary right I think there definitely a lot of research to be done how do you do visual

3:22:03 reasoning and uh language reasoning at the same time and in fact this is what humans do right so humans can imagine Futures but also humans can reason about future in in in words right and uh all that obviously is beneficial thank you for the great presentation I have F questions so um the chat box of the L to respond to the user to indicate how does the vehicle thinking pretty interesting idea uh just thinking about um instead of just responding to the user uh will you possible that the user pomping will potentially affect the driving of the vehicle so for for example if the user prefer driving in the highway instead of local row prefer to driving in the middle of the lane instead of the Caple Lane and the right most L would that make it possible to interact between the user and the vehicle itself yeah yeah yeah and uh so so I'm going to publish the slides and uh there is a list of papers uh and some of them actually

3:23:08 explored that exact idea not well not exactly the example that you're saying but the idea of instruction following idea of interaction with user uh and specifically those papers that focused on llm uh part not the VM part because unfortunately I mean still we we we still don't have good papers on VM agents right on visual language visual agents but there are papers exploring exactly those ideas yeah and answer is yes it's it's possible people demonstrated that and yeah hopefully that that's the future right you can change uh the style of driving say Okay I want to drive more maybe safely or aggressively or stuff like this right you can even instruct okay you have to go left or right and um all permutations of that thank you so much I'm really looking forward to the papers thank you all right let's thank again all to the great chat and and uh um for the last talk uh we have alah who is a head of AI research at wave uh she

3:24:26 will talk about uh the future direction of in atom Drive in hey it's all yours can you hear me yeah okay great uh and can you see the slides because I can't see this yes okay great that's amazing uh welcome everybody to this final session it's amazing you made it my name is elaha and I'm head of AI research at fa and I'm trying with this last session first and recap everything that was discussed today in very briefly and then discuss also very briefly at the end the advancement challenges and the future direction for the end to end autonomous Driving Systems uh the agenda would be uh first we will go through why end to endend compared to modular systems and then uh what was the advanced and Technology up to here we will go through the simulators word modeling language for self-driving and at the end we will discuss the directions and the challenges for the future research some of those will include like data scale

3:26:06 safety memory efficiency and so on uh the first question is the why end to end auton driving uh this diagram was uh shown few times already today I go very briefly into that uh the traditional autonomous driving which also we know them as a modular uh approach or AV 1.0 they are combined of multiple modules uh they have perception uh planning prediction localization and um many more modules that they are all working a stack backto back with the hand Engineering Systems to make the autonomous driving work this this uh these systems uh already they are very complex each are coming with um uh more like each component comes with with its own error that end up to have uh accumulative error or compound error effect at the end of this uh decision making process which leading to be a non-scalable solution and it's uh constrainted with the hand engineering rules and they are Rel their Reliance heavily on the HD Maps make them also

3:27:34 reduces their applications because many roads are changing over the times every single day the roads even in the uh urban area in the cities are changing and then new rules are new roads are added and then you are driving out of Roads not everywhere that very frequently the maps are getting updated based on that in AV uh 1.0 uh as I mentioned already the individuals modules uh are coming with the errors from individuals modules are compounded over the time creating a sign significant system errors and also the safety of these systems at unknown situations is still it as question although later we will come back to this Concepts that maybe it's in the community it's more uh accepted that they are more interpretable because you know what should be the output of each module but how the modules all together end up to interact with the word is still unknown and the safety scenarios are limited

3:28:49 there in contracts we have endtoend ai systems for or AV uh 2.0 that we are combining all these modules into one neural networks and then we let the system to decide and optimize for these errors that every individual uh modules in the AV 1.0 has to work with this approach eliminates the need for intermediate representations and allowing for direct optimization for the end results as I already mentioned and it's more scalable something that we haven't touch it yet these both systems they need uh safety subsystems uh because of regulations and also the public trust that later uh at the end of the session we will go that through that why end to end systems these systems are highly portable because they um because they use uniform computation structure they can easily adapt to new uh kind of Hardware settings and also uh vehicles and simplifies development with fewer modules which is a huge advantage

3:30:13 and achieves better performance compared to the these traditional human coded systems or rule based uh Design Systems this makes them uh compling choice for future autonomous driving Solutions and that's why the whole tutorial was uh around the end to endend systems uh one very important things we need to remember is that uh perception is crucial for our decision making but in the real world and real challenge it lies in the decision making process itself embodied AI must generalize to new unseen situations and adapt to different vehicle types and geographies this is what it's coming at the decision making process although the system can realize the traffic science and many other uh maybe objects but still the decision making how to interact with this Dynamic world and evolving world around it that's the major challenge uh this challenge itself requires robust decision making capabilities that go

3:31:32 beyond a simple environment perception uh this uh coming from a very nice uh survey paper that today was briefly discussed I want to touch it back be because we can after uh establishing okay we have AV 1.0 and 2.0 and 2.0 comes with many more advantages over uh the first modular systems approaches the conventional approaches now how we can test and verify and train these systems we can use real word data we can test them or even use the close loop data and we can do this verification and validation over open loop there are so many challenges some of them we already mentioned it was already touched one things that I want to very espe specify here is that interpretability is different than explainability and still it's a question that do we need to that the systems to be interpretable or we want them to be just explainable interpretability maybe goes more at the uh like the level or modules more than

3:32:55 just the whole process if the system make any decision and can explain the root cause of that decision and some of the future trends that uh it's like zero shot or few shot learning why we need that because every day we might come with a very new scene and scenario that nothing similar to that was already included in the data uh base that the models were trained and these few shot learnings are very useful imagine you are driving in Europe and then you go to u to USA or in UK and then USA or the other way around you can quickly adapt yourself from right hand side driving to the left hand side driving and this capabilities are coming with the few shot learning or zero shot learning or even the traffic signs from one country to the other country a slightly might change and if you just train your models to be uh to learn based on this memorization or uh I uh recognition of the traffic sici

3:34:07 perception modules that is coming with AV Point uh 1.0 then this might be very limiting our Drive driving system to just one country uh let's now recap some of the rec recent advances and technology in the field that was already discussed and uh this includes generative models for trajectory prediction data Centric approaches for continuous Improvement and integration of uh language models into the Driving Systems uh the generative models they are coming with the many advant pages uh this includes text and video models trained unsupervised video and at the internet scale text Data very huge amount of knowledge is already they're available and this is coming to us how to use them and Leverage The is knowledge from the textt into our decision making process these models show em uh emergent reasoning capabilities and enhan planning with in the combination with the word models and the video

3:35:33 Generations uh High uh Fidelity simulators are crucial why they are crucial for different aspects uh it's coming from the training generating the AG cases which is very hard and scarce to capture in the real world or sometimes they are very dangerous and we canot even try to capture those data we should avoid as much as possible and in other cases because they are so rare then there is uh almost impossible to dve or you need to drive Millions uh of kilometers to capture these scenarios and then the simulation comes handy to generate for the training purposes also for evaluation and it goes from open loop that has the the model output doesn't uh interact with the simulator and doesn't impact the state of the simulator to the Clos Loop which the model interacts with the simulator and the state is changing over the time based on the action uh word model was John Luca did a great job went through the many papers

3:36:50 I'm going back to a basic definition of it a award model is generative models that predict future Estates based on the current estate and action uh and these models are valuable for dynamic and uh reactive simulations something that at wave we are using at the moment and also they are very useful for what uh creating this counterfactual experimentation something that they are very useful again for the safety scenarios for uh checking if the action and uh language are already grounded and many other use cases are coming with the counterfactuals and then the modelbased reinforcement learning and search based planning these models are key to advance uh advancing autonomous driving technology and they their use cases is going beyond that uh this was already also shown but uh for re-emphasis on uh how like how the field is growing soap fast from two within a year almost how many papers are coming in this field and still there are

3:38:10 so much room for improvement and I hope this uh tutorial is just encouraging you to pick some of the topics some of the challenges and unresolved uh problems and the start working on them the other uh concept that uh it was just before this session was discuss is the multimodality integration into the driving system U one of these very useful modalities is the language because it comes with a huge uh knowledge which you we can use the internet scale data for create for acquiring that knowledge and now we have Vision language action models that they are interacting and they can use for different purposes for making the the system more safer more trustable because they can explain their actions and also to making more informed decision making and also coming with zero shot uh inference uh explainability was one of the reasons behind the language models uh there are other opportunities also

3:39:28 coming with the language in autonomous driving one of them is the intelligence this huge knowledge that is comes and the other one is the trust and safety if the model can reason behind its action not just taking an action based on some perception and planning but can resent like what's going on uh around uh me in the environment and how every other agent around me is going to interact with me and react even maybe to my action then this reasoning if it's there then the public can trust such a model more than the others this is a great SU summary of uh the differentiation between the AV 1.0 and AV 2.0 uh the a 1.0 is Rule based it's coming with many many modules for perception prediction uh planning and also the Reliance on HD Maps they are not robust Tod and also they are not scalable and because of HD Maps the their application is very limited to the root the to the routes that already we

3:40:47 have the HD maps for the data D the av2 2.0 has also data driven approach and principle based approach the difference is that for principled approach we have multimodality and then we use word models and so on this uh approach is com it's uh comes with the knowledge from the whole internet it's a real world and also the reasoning behind interaction with the real world physics and the prediction it's enable zero shot generalization and we can make use of Chain of Thought reasoning and wave it's already pioneered in using the principle based approaches many of our papers and blog post is coming such as lingo 2 that is already in the same era now looking ahead we we already have many challenges that we have uh had some Advanced uh advancement in those although none of them are fully solved uh but there are coming with a few more challenges this includes diverse real world scenario managing vast data scills

3:42:09 that will come or we need in ensuring safety making quick decisions and optimizing efficiency unique challenges that we are in the real world deployment for autonomous driving uh facing at this time they include processing massive data uh this massive data could be uh handling uh diverse and unpredictable like we don't know how much more data we need to capture to handle those and at the same time every single day and every deployment that we have on the road it brings more data and how we are going to train how we we are going to manage and how we are going to update the models based on this new data that's also another aspect therefore there are two different uh Direction with handling uh huge data sets data scales um also the the the safety concerns that I will discuss briefly optimization and uh the decision making these uh roads and the interacting with the all Dynamic agents in the road even the static object like

3:43:32 the traffic signs they are a static traffic lights but they are changing in a fraction of time and then the system needs also to interact uh in a fraction of second or even less understand that to be in a safe condition uh we need a lot of data why because every say every single day we might come with a novel combinations such as this one that it acts like a mirror that is showing instead of looking like a truck or another vehicle it could act as a mirror or a rare object on the road it could be any animal or any unknown even object and then people are in that area creative they can wear a stop sign or any other traffic signs or they can dress up like a cone and then you this might be uh very confusing for the systems that only based on the perception and planning is only coming from the perception based uh knowledge not a more reasoning and uh decision making process that it's beyond that uh there are also there is uh a

3:44:52 very big challenge with AV Point AV 2.0 is that inter the explainability it's such a blackbox but in general all deep neural networks are a blackbox there are millions of parameters and then explainability of these systems are really hard planning and prediction also uh the fit the the networks that we are training right now they are scared poorly with respect to the input output temporal length we don't the the systems they don't have a memory how we can add technical issue again um okay so we can't hear you now okay uh but it resumed okay okay great uh yeah one of the other things is that with the uh input output temporary uh lengths the networks they don't escale or scale very poorly we need a memory how to add memory to the systems this is not only for endtoend system is for even the modular systems we have this problem how to add Common Sense reasoning because of all these unpredictable scenarios that might we

3:46:13 might face and how to add uh change of rules or add H case without retraining everything from a scratch this slide also was uh shown today and this is very useful because when we are dealing with age cases we need way more data to collect than to capture any of those rare cases but can we with the generative models bridge this Gap with what we need for the training and what we can uh capture in the real world scenarios this is another uh area of the research that could come very handy but at the moment still all these simulation based uh training at least they are suffering from the seem to real Gap that is there memory was already mentioned we as a human we have it's coming from Daniel Cohen and this uh hypothesis that brain is involved through the time and we are using two subsystems of the memory one of them is called system one is uh they're responsible for 95% of our uh decision making reasoning and

3:47:37 interaction with the world this 95 is an autopilot and doesn't need that much reasoning in the brain to make any decision but system two is rational thinking is much slower in the uh time scale and also it's more cautious we need to make reasoning more maybe level of reasoning to make any action and it's a more cautious way of decision making but why we need memory systems it's crucial for uh handling this long uh context the temporal context that we already mentioned for the scenarios that is needed it's essential for reasoning and planning in the complex scenario especially with handling the edge cases longtail scenarios and uh enables continual uh adaptation to this new knowledge that every single day the data are collected from the road as I mentioned if we collect every single day do we need to retrain everything from scratch because adding this new knowledge it's coming with other kind of

3:48:52 if we don't retrain or we don't uh train partially on the previously seen data it comes with the many more challenges like catastrophic forgetting that neural networks are suffering from and then if we want to get closer to uh AGM that one of the aspect is learning how to learn we need this memory system to deal with those situation uh the other uh very important uh Gap that already exists with all this advancement in the field that uh we have with the large language models with the word models and so on uh more larger and larger models are being trained with the more uh number of data points but at the end of the day for the embodied AI which is autonomous driving is one of those use cases we need to deploy on some agent that it's living in the world and interacting with the in with its surrounding and for this to be more sustainable it needs to capture to consume less energy but the energy

3:50:11 doesn't come only at the inference time even at the training like continual learning if we need continually up update the knowledge of these agents we need also to make them efficient but efficiency at the model level is one of the main challenging if we decide to go to much a smaller uh Hardware usage then there is a huge gap at the moment safety and re uh reliability of of the autonomous driving in general for both uh one and av1 and av2 are there and they are still unresolved there are uh several uh unresolved challenges there this includes managing the unpredictable nature of open Bo deployment handling the massive escale of the data from multiple sensors at any uh given uh point of time any of these sensors can fail and then the system needs to be reliable and be trustworthy that can in decide if the rest of the sensors are enough or at which case it can continue interacting with the word or just uh

3:51:34 stop working and ensuring all utmost safety for all Road users making rapid informed decision additionally optimizing for efficiency in uh C is crucial to maintain High system performance there is a lack of a standard uh standardized and in uh scenario databases for uh testing and verifying the AV systems and the uh deploy uh development cycle for autonomous driving in general is very slow lengthy and costly and also deploying uh under real world and testing for all cases is impossible because some of them are dangerous and unclear and varing uh regulation across re regions going from Europe to Asia to us there are many different rules that with different uh policy makers and Regulators are coming and it's really hard to handle all these different uh safety scenarios although there is no agreement on what is is called Safety and what scenarios the system needs to be tested so there is a new era of of

3:52:54 research that it's a still open question the driving uh environment complexity means number of scenarios uh for autonomous is virtually infinite and a study suggest that providing and uh ads is which is 20% safer than a human driver requires over 11 billion miles which is very huge and almost impossible to capture for uh very near future and uh training our models and testing them however uh simply driving many miles it's insufficient because a diverse range of sunar must be encountered to ensure safety the key challenge lies in finding the right scenario that uh represent the OD of the autonomous driving system determining if enough scenarios have been identified to guarantee system safety considering their probability of occurrence and challenges include uh assess uh assessing the safety risk associated uh associated with each of these scenarios and then after defining all these things then we need to Define what

3:54:23 does it mean a good behavior or pass fail criteria for autonomous driving in given scenario therefore this safety is an open field and also there is no consciousness over that and it needs more uh attention after research Community uh okay the I hope you enjoyed the tutorial and it was useful giving you some just hint what has been done and where you can find those uh papers and what is a still open question unresolved Direction and challenges at the moment and how we can uh tackle them uh we are hir hearing at wave and we are working on all aspects from simulation towards modeling uh language for driving efficient Ai and also the memor multiple memory systems uh different architecture based uh com very Advanced and very Noel and if you are interested of any of these areas we are hiring in London and uh Silicon Valley and Vancouver our new science office uh please uh go and check our website and also come and meet our

3:55:54 team at the boo Hi elah not sure sure if you can see me but this is K your colleague from T einhoven thanks a lot for your great presentation I have one uh one question if you look at the human brain it it has some modularity and so you have parts of the brain that deal with very quick visual reflexes to stay safe and you have parts of the brain that deal more with yeah high level reasoning now in this era of endtoend uh U autonomous driving do you think that there will be some modularity uh let's say a system of networks that can be trained end to endend but there is still modularity or do you think it's more it will in the end be just one big Transformer or whatever architecture hi thank you for the question uh yes brain also has some subsystems but none of these subsystems are trained individually and in isolation with the rest of the brain that's some differences that we are fa we are seeing with the AV

3:57:07 1.0 but a 2.0 could be a subsystems of different Ma that also it's called AV 1.5 even sometimes that still we think okay perception and everything should be trained back to back and in a end to endend uh kind of uh manner but still we can have this subsystem but AV 2.0 what we are hoping or what we believe is that there is emerging emerging sub networks that may be is interacting with the word some of those subsystem maybe uh lower layers are responsible for perception but we are not distinguished at what level this emerging of the perception should stop and the rest for the planning should start and so on yeah okay clear thank you for uh your answer all right let's s again hisory talk and uh yeah this is the end of the tutorial and and we will release a PO recording in our website so please check our website later and we also about Bo at 1712 so if you want to talk to our engineer Eng please to

3:58:32 our thank you so much thanks thank you L thank you he great talk

Questions about this episode

What is discussed in "23722 End to End Autonomy A New Era of Self Driving"?

In order: 0:00 Hong Kong University; 15:00 Gaia; 2:00:00 Dreamer; 2:15:00 Dreamer, Gaia, Drive Dreamer; 2:30:00 Dreamer.

What is mentioned in this ComputerVisionFoundation Videos episode?

10 people, works and ideas are mentioned, including Wave, Prism, Gaia, Sora, Dreamer, Hong Kong University. Each one on this page links to its own page with every episode that mentions it.

How long is "23722 End to End Autonomy A New Era of Self Driving"?

The episode runs 3 h 59 min; the transcript is about 35,310 words in 207 timestamped sections.

More from ComputerVisionFoundation Videos

• All analyzed episodes
• People mentioned on ComputerVisionFoundation Videos
• Topics mentioned on ComputerVisionFoundation Videos