NVIDIA's Marco Pavone on AV Safety, Simulation, and Halos

Source material

Transcript

[Music] Hello, and welcome to the NVIDIA AI Podcast.

I'm your host, Noah Kravitz.

If you've been in San Francisco, Phoenix, or a number of other cities recently, no doubt you've seen cars navigating the streets without anyone in the driver's seat.

Robo taxis and other autonomous vehicles are becoming a part of our lives, whether through prototype testing on public roads or driverless cars ferrying passengers across city streets.

Autonomous vehicles hold a lot of promise, but first and foremost, they have to be safe.

This is where advanced simulations, powered by AI, can help.

Here to explain the ins and outs of autonomous vehicle safety and simulation, and why it's so important to all of us going forward, is Marco Pavone, Senior Director of Autonomous Vehicle Research at NVIDIA and Professor at Stanford University.

Marco, welcome, and thank you so much for taking the time to join the NVIDIA AI Podcast.

Thank you for having me.

So, Marco, if you don't mind, could we start by having you tell us a little bit about what your role is at NVIDIA and Stanford as well, if applicable, when it comes to autonomous vehicle research, safety, simulation, and everything else we're going to talk about?

Sure.

At NVIDIA, I lead autonomous vehicle research, and our research spans a number of different research tasks.

First of all, we work on developing new architectures for autonomous systems.

We work on developing ED Foundation models that can empower the full development lifecycle, the full development program from the onboard stack, all the way to simulation, data curation, and even safety evaluation.

We work on enabling high fidelity behavior and sensor simulation, and we also work on developing tools to ensure the safety of AI-based stacks.

At NVIDIA, I focus primarily on ground robotic systems, namely self-driving cars, while at Stanford, I focus primarily on autonomous aerospace systems.

How fascinating.

We'll have to have you back to do an episode on the potential future of flying cars, putting those two things together.

So, in the introduction, I mentioned robotaxes.

It was something that came to mind for me as an easy way to kind of visualize what we're talking about.

But as I understand it, your purview and what NVIDIA is working on more broadly isn't just fully self-driving vehicles.

There's more to it than that.

Yes, the field of autonomous vehicles is very diverse, both in terms of the technology from semi-automated systems all the way to fully automated systems, like, for example, robotaxes, and also in terms of application domains.

Personal mobility, namely self-driving cars for people moving in cities, all the way to freight, transportation, autonomous vehicles for agriculture, for construction, and more.

Right, excellent.

So, let's start maybe by talking about Halos.

NVIDIA recently announced, well, I'm going to leave it to you, something called Halos.

Can you tell us about it and how it relates to AV safety?

Yes, of course.

So, safety obviously is paramount for autonomous systems.

They are safety-critical systems.

And Halos plays a very important role in this regard.

So, specifically, Halos is a full stack, comprehensive CPT system for autonomous vehicles that unifies safety elements from vehicle architecture all the way to AI models.

And in particular, Halos comprises hardware and software elements, tools, models, as well as design principles for combining them to safeguard AI-based and to end AV stacks.

The hardware and software, is this in vehicle, in the cloud?

Does it span both?

As I mentioned before, it spans everything from the vehicle architecture.

So, basically, the hardware and software goes into the car all the way to the development processes that happen in the cloud to be the safe AI system.

For example, Halos comprises the water-first CPT-assess platform for AI-based AV stacks.

So, this goes into the vehicle, and it also comprises dedicated MLOps workflows for safety data curation.

So, as you can tell, it's a very broad problem.

Very comprehensive.

Yeah, great.

Okay, so let's talk about AV safety.

This is one of those questions that I have a million answers to in my head as I ask it, but I'm sure none of them hit the mark.

But why is safety important for autonomous vehicles?

Well, autonomous vehicles are safety critical.

So, the consequences of a mistake, of course, can be catastrophic, including loss of life.

So, obviously, you have to make sure that these systems do not pose an unreasonable risk to society.

One, because this is the right thing to do, and two, because otherwise, they will not be accepted by society.

So, this is the reason why NVIDIA, who is developing its own AV program, but is also working on developing an ecosystem where multiple stakeholders can develop their own AV programs, is so invested in pushing safety from vehicle or the way to the cloud.

So, when we dig into this a little bit, what are some of the key considerations when it comes to ensuring AV safety?

Yeah, so I like to say that there is no silver bullet to ensure AV safety.

There is no single technology or single process that is going to make your system really safe.

Safety really has to characterize the full development problem from design time, making sure that, for example, we train the AI systems on safe driving behaviors all the way to runtime and deployment time.

So, once the system is deployed on the road, building monitors that, for example, keep in check the AI system or building guardrails, all the way to the iterative development of the system.

So, basically, how you learn from your deployments to increase the safety of the system throughout its lifetime, potentially over the years when the system is deployed.

As I understand it, and mine is very much a layperson's kind of working understanding, but one of the things, or perhaps the overarching thing that makes developing autonomous vehicles so challenging is that the conditions that they're dealing with are literally unpredictable.

Weather driving conditions, the conditions of the roads, a kid on a bike starting out across the street, all kinds of factors that you have to account for even if you can't actually imagine them ahead of time.

But can you talk about some of the common challenges that developers face at different stages of the lifecycle when developing AVs and how they make sure that safety, drivers, passengers, pedestrians, everybody, is prioritized?

You're exactly right.

So, by definition, an autonomous system is a robotic system that operates in scenarios that were not foreseen at the design time and requires some level of reasoning.

So, how do you ensure the safety with respect to conditions that potentially might be quite different from those seen at design time when the system was developed?

And as I said before, there is no silver bullet.

It goes down to fundamental principles of safety engineering.

So, making sure that we build a stack that is diverse, meaning that comprises components that are different and with overlapping responsibility.

It goes down to the principle of monitoring, meaning that you have to constantly monitor if you will the else of your system to make sure that, for example, it's not operating in situations whereby you cannot assure anymore the trustworthiness operation.

So, basically, the system needs to understand when it is outside its domain of expertise.

And of course, it boils down to the principle of testing and validation.

So, how do we robustly test the system in a way that covers as much as possible all the most important situations in my phase at deployment time?

Right.

And are there accepted standards or benchmarks, or how do you determine when a system is deemed safe?

So, the level of safety really depends on the product we are talking about.

Okay.

So, obviously, a semi-automated system whereby the human is in charge of monitoring the system itself has lower safety requirements than robotaxis.

Right.

The human can intervene if need be.

Yeah, exactly.

So, there is a continuum spectrum of the safety requirements depending on the level of automation and there are well understood processes to go from an analysis of the potential risks and consequences to requirements on the stack, for example, in terms of mean times between failures.

Then how basically you achieve those, again, there is no silver bullet.

It goes back to making sure that you have a safe design, for example, reflecting the principle of diversity, that you have a robust data set curation pipeline to make sure that you train your system on safe behaviors, to make sure that you have a robust runtime monitoring pipeline that keeps the system in check, as well as a very strict test regimen to test your system using a combination of both real world data.

You still need that even though that is expensive.

And also simulation data, and this is something that is becoming increasingly available to developers, which I think is very exciting.

So, to talk a little more about simulation, I mentioned in the introduction, prototype vehicles testing on public roads, city streets, and all of that.

But it sounds like that's not the only way to test and validate the safety of these systems.

How do simulations work in this context?

I mean, simulation has always been the holy grail of robotics more broadly.

If we had a perfect simulator, robotics development would be dramatically accelerated.

And of course, perfect simulator doesn't exist, at least yet.

But in the past two, three years, the robotics community broadly has made a number of breakthroughs in the field of simulation, both in terms of how we simulate the environment from a sense of realism standpoint.

So basically, rendering images and environments in a way that is ultra realistic.

It will be very hard for you to tell the difference between a computer-generated image and actually a real image.

And also in terms of technologies to replicate how other agents, for example, bikers or pedestrians would behave on the road.

So all these technologies together have basically enabled the simulation capabilities that even two, three years ago were not even imaginable.

That's been quite a few years in this industry.

Yes.

And now simulation is more than ever part of the development process, for example, for testing.

If you have a new design, you want to make sure that you improve according to the number of metrics in a number of different operational design domains.

Simulation is a great tool for that.

Using simulation for validation is still something that is a bit tricky because it means that you're not just interested in directional information.

That is, whether something is better than something else, but you're also very interested in the absolute numbers of your metrics.

And so the question is, how do you trust your absolute numbers?

For example, we've been doing research on this topic on developing statistical methodologies to provide confidence bounds on the metrics that are generated by the simulator to help developing a validation case that comprises both tests of real data.

As I said before, you need that.

I don't see there is a way around it, but also simulation data thereby dramatically decreasing the cost of validation.

Are there specific, I guess I'm thinking of both points in the development process, but then also situations that a vehicle, I'm thinking about it as a driver, but a vehicle might encounter, that simulation has proved particularly adept at?

So simulation helps both with respect to testing against nominal conditions, like for example, whether you have good performance when negotiating an intersection and so on and so forth, but also with respect to anomalous conditions that might be very hard to see in real life.

And one of the recent work that we had, that I think is very much related to your question, was to use large language models to recreate crash scenarios that have happened in real life.

So basically mining the police reports that have been written over the years because of crashes, and then using a large language model to interpret them, adds a little bit of generative AI, and then recreating scenarios that have been very hard to recreate just by asking designers and artists to do so.

So to recreate those scenarios, but in a way that are still plausible.

It's very easy to simulate very challenging scenes.

The question is, are those things that can really happen, are they plausible?

But basically grounding the generation in crash reports, you have that grounding on those scenes.

Oh, that's fascinating.

And we've seen how actually these scenes can improve in both testing and validation.

So this was a long answer.

The short answer to your question is yes.

We've seen situations where we were able to generate scenarios that would have been very hard to generate otherwise without our AI-based workflows, and they have helped with improving the performance of the system.

Right.

That's fascinating.

It's one of those things that when you look at it, you're like, oh, that's such a simple idea, makes so much sense, but very clever.

It's interesting.

How does simulation, AV simulation for autonomous vehicles, how does that differ from traditional automotive simulation or even other methods of simulation?

Yes.

So AV simulation has some features that make it particularly challenging, and some other features that actually make it simpler than other domains, like, for example, robotics.

One feature that makes AV simulation, particularly in the context of high level of automation, so semi-autopilots or robotaxis, is that you have to reason about the interactions with other agents on the road.

Human drivers and pedestrians, bikers, animals possibly.

Those interactions are very difficult to model because essentially you're modeling a decision making agent, and you want to do that in a way that is highly dynamic.

So basically it reflects the physics of an interaction, but also the behavioral dynamics.

So how, for example, a biker might respond to what an autonomous car is doing.

It's a sort of a ballet between the two agents.

So that part is very hard, but it's very important if you really want to push the utility of a simulation, particularly for higher levels of automation.

For more basic levels of automation, like, for example, collision avoidance or lane keep assist functions, you don't need, you don't have such a requirement on modern interruptions, for example, which make it simpler.

There are other parts, though, that make AV simulation easier.

Like, for example, the dynamics of a vehicle, they are relatively simple and well understood, at least as compared to the dynamics of a humanoid robotic system that is, for example, grasping an object and then you have to reason about friction.

Maybe the object is a software to reason about the formation.

So they're very hard, and it is not something that you have to deal with in AV simulation.

So I think there are features inside that make it more difficult, other than make it a little bit simpler, but definitely it's very hard.

Yeah.

How did developers navigate testing using simulations to test in controlled environments versus uncontrolled environments?

And you were talking about, I almost was going to ask you if you have to kind of weave some unpredictability into these other agents, imagining, well, is it a company, somebody who's used to being on a bicycle, or is it a kid just learning for the first time and will react differently?

But obviously controlled environments, uncontrolled environments are both important parts of the processes, but how do they relate and how does the use of simulation kind of differ in the two?

Yeah.

So as I mentioned before, simulation helps both in terms of simulating nominal behaviors, which is still important, for example, to test whether a new autonomy stack is potentially regressing on some metrics, like for example, how well and how comfortably you're driving on a highway.

But at the same time, you're also interested in more rare cases, corner cases.

And here is where you leverage different technologies, like for example, for sensor simulation, there have been significant advances in the past two, three years on the topic of a newer reconstruction.

So basically reconstructing a scene potentially from a different point of view to allow for new trajectories that a vehicle can execute in a simulated environment.

To back up just a step, Marco, I apologize.

But when you say sensor simulation, does that refer to simulating based on inputs gathered by sensors or is it something else?

Oh, I mean, basically recreating a scene as it will be perceived by a camera or a ladder and so on.

Okay.

So basically generating, for example, images that are synthetic, but that are very close to the ones that your cameras would see at deployment time.

Okay.

So what I was saying is that there are technologies that allow you to reconstruct very faithfully a scene, but of course, a reconstruction technology reconstructs a scene that was observed in the real world.

So basically it's bound to reconstruct whatever has been seen.

So that's where other new technologies related, for example, to video generation, like for example, the Cosmos model that NVIDIA has produced come to the rescue or help because they provide the capability of generating new scenes, something that is completely different from what has been seen in the real world, maybe in the conditional fashion.

So these models can be conditioned, for example, in text prompts to create the scenarios that are deemed a particular interest.

Like for example, I want to simulate a kid that is chasing a ball on a crosswalk.

And now we have the technology to do that.

Right.

And I would imagine the models and these systems have guardrails that, to speak to what you mentioned earlier, that kind of make sure that the scene being simulated is something that could actually happen in the physical world and not out of those.

Or at least plausible.

Or at least plausible.

Right.

Plausible is better work.

Yeah.

That's one of the reasons why there is substantial effort at NVIDIA and in the community in general to make sure that the simulation is so called the controllable.

Right.

Or in other words, you can abcabulate to it.

Right.

We've talked on the podcast recently about digital twins and the role of digital twins in industry and industrial AI and being able to simulate, well, all kinds of things happening in a factory environment, say, before actually trying to construct or deploy them physically.

How did digital twins play a role in AV simulation?

Well, in a way, an AV simulator is a digital twin of a city.

An interesting digital twin, because as I said before, differently from other domains, for example, the domain of industrial robotics or the aerospace domain, in the urban digital twin, you also have to model humans.

Right.

But in general, yes, digital twins are revolutionizing a number of fields, including aerospace, which is my other passion, by providing unprecedented tools for designing aircraft systems completely or almost completely in simulation, along with the control laws to control them.

And actually, I do have some projects around those lines that are also quite exciting.

Not to take us off course, but I don't know if you can speak to this briefly.

It might be an unfair question.

But how different is it, running simulations, working on AV cars, vehicles on the ground, as opposed to working on, I don't know if you call them AVs, but in the aerospace industry?

Well, in aerospace, depending on what you want to do, so depending on whether you want to test a new design, or maybe you want to test the new control laws, or maybe you want to test operations, of course, you have different type of simulators.

But in general, one important aspect of the simulator is that of accurately modeling the aerodynamics of the system, which particularly in the case of aircraft design has to be extremely accurate.

And today, thanks to advances in AI, we can run simulations where we have very detailed simulation of the airflow around the aircraft in a fraction of a time that was even possible a couple of years ago.

And interestingly, I do have a project whereby we try to develop a sort of mini digital twin that can even be run online and used for the purposes of control, to control highly flexible aircraft.

So these are basically unmounted aircrafts that tend to be much more lightweight than regular aircraft.

And so they have the formal mode modes, meaning that the aircrafts can actually, the wings can actually deform a little bit.

So you want to have a digital twin that allows you to predict how your body might deform as a consequence of your control actions.

Well, now we have the technology to do that.

Yeah, amazing.

Speaking with Marco Pavone, Marco is the Senior Director of Autonomous Vehicle Research at NVIDIA and also a professor at Stanford University.

And in addition to, well, as we've been talking about, in addition to his work on autonomous vehicles, cars and the like, also working in the aerospace realm at the same time.

And Marco, you alluded to this, well, throughout the conversation, but just now again, the past few years have just been breakthrough after breakthrough in AI, in generative AI in particular.

But how are these breakthroughs in generative and also physical AI influencing AV simulation now?

In multiple ways.

So breakthroughs in terms of reconstructing environments, for example, visually, in a way that is very difficult to distinguish from the real world, at least to a human eye.

Breakthroughs in terms of generating completely new scenes and scenarios by harnessing, for example, the power of video generation models.

Breakthroughs in terms of using language models or other foundation models to be able to parse human knowledge in terms of potential crash scenarios to recreate those in simulation.

And then also breakthroughs in terms of modeling the behaviors of humans in simulation.

So it's not just one single breakthrough, it's the convergence of multiple breakthroughs that interestingly have happened in the past really few years.

It's interesting listening to you talking and thinking like, "Wow, you see so much of the world when you take a drive in a car somewhere and all of these different elements you're talking about needing to simulate to really reconstruct a situation and test and validate for safety."

You see it, the natural world, the changing road conditions, the kids and the animals and all of that.

So it's fascinating to hear you talk about the ways those come together by virtue of these AI breakthroughs that allow us to model these different things.

Yeah, and interesting what you're saying because one of the things that these breakthroughs allow you to do is to tap into highly heterogeneous data.

Meaning that traditionally people were using data collected by a fleet of vehicles owned by a company developing the system.

But now with this new foundation models like video generation models, vision language models, better language models and so on, we can actually tap into internet scale knowledge.

For example, we can use videos that have been recording through dash cams maybe by taxes.

And all of a sudden, instead of relying only on the data gathered by 1000 vehicles that are part of your fleet, then you can collect an enormous amount of driving hours.

Not only incredible, not only in terms of the magnitude, but also in terms of the diversity because of course humans across the world are driving.

So you're able to collect data across the world simultaneously.

So this is yet another opportunity that has been unlocked by the most recent AI breakthroughs.

So along those lines, what are some of the other potential opportunities you see that world foundation models are opening up for AV simulation?

So far we have discussed the use of world foundation models mostly for the purposes of the simulation.

And that's absolutely a very important use case.

But then you can think a bit more.

So if you have a model that can really predict what will happen in the future, or at least in the next few seconds with a high level of fidelity, both in terms of how the environment will look like and in terms of what the other agents would do, it means that your model has implicitly learned a very powerful model of how the world works.

And so you can use that understanding as the basis for policy construction for basically building the AI that goes into the vehicle.

So I think that another opportunity that is right now still under development is to distill down these extremely powerful knowledge in a way that can be used also as part of the onboard stack.

Media is working heavily on it.

And of course, other people in the robotics community are also working heavily on it.

This is going, I think, to be a lead potential to another substantial breakthrough.

Right.

Amazing.

So you've touched on this a little bit in the conversation here and there, but are there any research projects that you've worked on recently or are still working on now related to safety and SIM that might be helpful for the conversation for the audience to hear about?

So yeah, multiple projects.

So one that I'm particularly excited about and that I alluded to at the beginning is how do we best leverage simulated data and real world data to provide metrics with rigorous confidence bounds so that we can really trust them and use for validation purposes.

Then we are doing research on how to curate data to ensure that the data is safe so that you only learn on safe behaviors.

And you might think that defining safety is easy.

You don't want to collide.

But actually, safety is much more nuanced than death.

So think about how you may want to teach a kid how to drive.

You start saying, well, you shouldn't have put yourself in that situation because if the other driver were to be attentive, then you might have collided.

So how do we capture all those potentially dangerous situations and distill them down into a data set that we can use for data curation and testing?

So this is another area that is very interesting.

And the third one is how do we automate safety evaluation?

As I said, the safety assessment, understanding what the reason why reasonable risk is, is complicated.

Of course, we can rely on human judgment, but that doesn't scale.

So how we can actually train the separate AI models that can help with the safety evaluation.

So that's another interesting project we have coming from going that we hope will bring real value soon.

And so along those lines, looking ahead in the next couple of years, five years, maybe it's more like 10 years, where do you see autonomous vehicles and specifically talking about safety and simulation?

But you can go more broad than that.

Where do you see the technology, the industry, society's relationship with AVs headed?

So I think that in terms of semi-automated vehicles and largely automated vehicles with a high confidence, this technology will be pervasive in our society within the next five years.

We already have companies that are productizing these systems, multiple companies that are doing so, and it's delivering real value.

So I'm quite confident in this statement.

In the case of robotaxis, the estimate gets a little bit more difficult in the sense that is in a situation whereby we do have a technology proof.

I mean, we do have robotaxis, for example, from Waymo that are driving around in San Francisco without a safety driver very safely.

And so that is a model, if you will, of engineering.

But the question is, when will this technology be able to scale up to the level that also becomes a profitable technology?

So I would say it's more of a business question at this point than a technology equation.

Of course, to become profitable, you also need to have technology improvements.

That is a bit harder to predict, but given the progress in the past few years, definitely, and especially given the progress by Waymo and others on the point of robotaxis, it stands to reason that within the next five to 10 years, many cities in the United States, in China, and elsewhere will have robotaxis installments.

To what extent personal mobility will be taken over by robotaxis is still very hard to predict.

Not just technological, it's also a business consideration.

And of course, there are many forces at play.

Just a personal question, and perhaps I should know this, but I've been behind next to some of those Waymo vehicles in San Francisco.

The apparatus on top that's spinning constantly, is that a camera or what is that?

So Waymo basically, as basically all other highly automated vehicles, has a range of sensors.

So cameras is one of them.

But then another important sensor is referred to as ladders.

So ladders sensors are laser-based sensors that measure distance as a function of time of flight.

And so they allow you to very accurately characterize the distance from objects in a scene and also provide the redundant way of a pursuing environment, which as I said before, is one of the key principles of redundancy.

It's one of the diversity is one of the key principles of safety.

So if, for example, a camera has a problem, you have a ladder that might supplement and pitfall to the camera.

So yes, a Waymo system has a bunch of ladders, sensor, camera sensors, and so on.

Got it.

Marco, this has been fascinating and so many different elements to think about.

And for listeners who want to dig in further, would like to know more, where can they go online on the NVIDIA website somewhere else to find out more about AV safety, maybe about Halos, any of the other things you've been talking about?

Yeah.

So at the last NVIDIA GTC conference, we had an AV safety day.

So it was a program, a three hour program where we went in great detail on all the different safety elements that comprise Halos.

So the recording of the presentations along with the slides are available on the NVIDIA GTC webpage.

And then there is NVIDIA Halos webpage that the people can refer to in order to learn more about Halos.

So just type "Vidia Halos" on your favorite browser, and then this page will pop up.

Easy enough.

Marco Pavoni, again, thank you so much for taking the time to speak with us.

Really, just I keep saying this, but a fascinating look into all of the different things that go into developing these systems.

And as you put it, all of the nuances that go into defining what safety is.

Right.

Thank you so much for having me.

[Music]