Transcript

Agentic AI MOOC | UC Berkeley CS294-196 Fall 2025 | Autonomous Agents by Peter Stone

From Berkeley RDI Center on Decentralization & AI · Episode page · Watch on YouTube

· 1 h 2 min · 9,755 words · 92 timestamped sections

This is the full, timestamped transcript of "Agentic AI MOOC | UC Berkeley CS294-196 Fall 2025 | Autonomous Agents by Peter Stone", an episode of Berkeley RDI Center on Decentralization & AI published December 1, 2025 (1 h 2 min, about 9,755 words). It mentions 24 people, works and ideas, including Peter Stone, GT Sophy, Sony AI, Gran Turismo, Brad Knox.

Who speaks in this episode?

What is discussed, and when

  1. – 10:00: Peter Stone, RoboCup, Adam Klivans, Noam Brown
  2. – 20:00: Nature, GT Sophy, Sony AI, Gran Turismo
  3. – 30:00: Brad Knox, CoRL, GT Sophy, Roberto Martín-Martin
  4. – 40:00: GT Sophy
  5. – 50:00: GT Sophy, Sony AI, Gran Turismo, Polyphony Digital
  6. – 1:00:00: GT Sophy, Gran Turismo, Nature
  7. – 1:01:45: Sony AI

Notable quotes from "Agentic AI MOOC | UC Berkeley CS294-196 Fall 2025 | Autonomous Agents by Peter Stone"

  • So what I'm going to do in this session in this talk is, as advertised by the title, I'm going to talk about my view of autonomous agents, and in particular a broad view, not…
    about Peter Stone · at
  • This is an organization that spawned the Machine Learning Lab that's headed by Adam Klivans and Sanjay Shakkottai.
    about Adam Klivans · at
  • The goal of RoboCup, the Robot Soccer World Cup, is to create a team of humanoid robots that can beat the best World Cup champion on a real soccer field by the year 2050.
    about RoboCup · at
  • We had a paper on the cover of Nature, where we created reinforcement learning agents that were able to drive faster than the very best human drivers at the game, Gran Turismo, which is a very physically realistic simulation environment.
    about GT Sophy · at
  • For-- basically, as I said here in the tagline, and this was led by Alice Xiang It's the first globally diverse, consensually collected fairness evaluation data set for a wide variety of human-centric computer vision tasks.
    about Alice Xiang · at
  • And there's a system that was developed by a former student of mine, Brad Knox, who's now faculty here at UT Austin, that was a system called TAMER, Teaching an Agent Manually via Evaluative Reinforcement.
    about Brad Knox · at
  • And one of the key lessons here was that TAMER learns more quickly than autonomous learning from straight reinforcement learning.
    about TAMER · at
  • So the motivation here is, and I should say this is the PhD thesis work of Jiaheng Hu, shown here, and he's co-advised by myself and Roberto Martín-Martin.
    about Roberto Martín-Martin · at

Who and what is mentioned in this episode?

People

  • Peter Stone — Person · Main speaker · 3 mentions · first at
    Delivers the lecture framing autonomous agents broadly (sensing/acting loop) and surveys his lab’s work across robotics, multi-agent systems, and reinforcement learning.
  • Brad Knox — Person · Referenced researcher (former student) · 3 mentions · first at
    Cited for TAMER, an early reinforcement learning from human feedback (RLHF) system, used as a key example of human-in-the-loop learning.
  • Alice Xiang — Person · Referenced research lead · 1 mention · first at
    Credited as leading a Nature paper introducing a globally diverse fairness evaluation dataset for human-centric computer vision, used to discuss ethical dataset construction and bias evaluation.
  • Roberto Martín-Martin — Person · Referenced collaborator (co-advisor) · 1 mention · first at
    Identified as co-advisor (with Stone) of Jiaheng Hu on a CoRL paper about real-world robot reinforcement learning using a simulation-pretrained latent action space.
  • Noam Brown — Person · Referenced guest/expert (prior session) · 1 mention · first at
    Mentioned as a previous speaker in the course who covered multi-agent systems, helping situate Stone’s talk as emphasizing embodiment/robots.
    So you had from Noam Brown, you heard about multi-agent systems.
  • Adam Klivans — Person · Referenced campus leader · 1 mention · first at
    Named as a leader of UT Austin’s Machine Learning Lab (spawned from an NSF AI Institute), providing institutional context for Stone’s environment.
  • Sanjay Shakkottai — Person · Referenced campus leader · 1 mention · first at
    Co-mentioned with Klivans as heading UT Austin’s Machine Learning Lab, reinforcing the institutional ML foundations around Stone’s work.
    This is an organization that spawned the Machine Learning Lab that's headed by Adam Klivans and Sanjay Shakkottai.

Companies & orgs

  • Sony AI — Company · Business focus · 4 mentions · first at
    Research division of Sony focused on AI, responsible for developing GT Sophy and other AI innovations.
    We had a paper on the cover of Nature, where we created reinforcement learning agents that were able to drive faster than the very best human drivers at the game, Gran Turismo, which is a very physically realistic simulation environment.
  • Texas Robotics — Company · Affiliation · 1 mention · first at
    Organization at UT Austin focused on robotics research, mentioned in the context of the speaker's work.
    I was the founding director of Texas Robotics.
  • Polyphony Digital — Company · Affiliation · 1 mention · first at
    Developer of Gran Turismo, mentioned in the context of GT Sophy development.
    But by doing this within the Sony ecosystem, we were able to partner with Polyphony Digital and Sony Interactive Entertainment for the PlayStation group to have access to a…
  • Sony Interactive Entertainment — Company · Affiliation · 1 mention · first at
    Division of Sony responsible for PlayStation and Gran Turismo, mentioned in the context of GT Sophy development.
    But by doing this within the Sony ecosystem, we were able to partner with Polyphony Digital and Sony Interactive Entertainment for the PlayStation group to have access to a…
  • Waymo — Company · Case study · 1 mention · first at
    Company working on autonomous driving, mentioned as part of the broader context of autonomous agents.
    As you know the car companies and companies like Waymo and Zoox are putting a lot of money into making this a reality.
  • Zoox — Company · Case study · 1 mention · first at
    Company working on autonomous driving, mentioned as part of the broader context of autonomous agents.
    As you know the car companies and companies like Waymo and Zoox are putting a lot of money into making this a reality.

Products & tech

  • GT Sophy — Product · Case study · 13 mentions · first at
    AI agent developed by Sony AI that outperforms human drivers in Gran Turismo, demonstrating advanced reinforcement learning techniques.
  • Gran Turismo — Product · Case study · 4 mentions · first at
    Highly realistic racing simulation game used as a platform for testing and deploying GT Sophy.
    Where we created reinforcement learning agents that were able to drive faster than the very best human drivers at the game, Gran Turismo, which is a very physically realistic simulation environment.
  • TAMER — Product · Case study · 1 mention · first at
    Early system for reinforcement learning from human feedback, precursor to modern RLHF techniques.
  • Deep Tamer — Product · Case study · 1 mention · first at
    Extension of TAMER for learning from image input, mentioned in the context of human-in-the-loop learning.
    Then collaborating with Garrett Warnell, pictured on the right, we had an extension to video or image input, a system called Deep Tamer.

Concepts & topics

  • One Hundred Year Study of AI — Concept · Referenced work · 1 mention · first at
    Long-term study on the future of AI, mentioned in the context of ethical AI initiatives.
    We also have an ethical AI system on campus that's inspired by the One Hundred Year Study of AI that I've been a leader in.
  • FHIBE — Concept · Case study · 1 mention · first at
    Ethical and globally diverse image benchmark for evaluating fairness in AI models.
    And so tried to start with FHIBE, tried to establish an industry standard for AI ethics.

Places & events

  • RoboCup — Event · Case study · 4 mentions · first at
    International robotics competition focusing on robot soccer, serving as a testbed for autonomous agents research.
  • DARPA Urban Challenge — Event · Case study · 1 mention · first at
    Historic competition for autonomous vehicles, referenced as an early example of autonomous agent research.
    I had a car in the DARPA Urban Challenge, the one pictured here.
  • CoRL — Event · Referenced work · 2 mentions · first at
    Conference on Robot Learning where recent research on real-world reinforcement learning was presented.
    With regards to embodiment, I'm going to take a little bit of a deeper dive and show you a paper from just-- it was presented at CoRL just about a month ago.
  • RoboCup@Home — Event · Case study · 1 mention · first at
    Competition focusing on general-purpose service robots, mentioned as inspiration for research.
    So there's a competition called RoboCup@Home, trying to create robots that are generally capable in this way, and that's been very inspiring to my research as well.

Media & podcasts

  • Nature — Media · Referenced work · 6 mentions · first at
    Prestigious scientific journal where research on GT Sophy and other AI advancements were published.
    So this was the first time-- the reason it was picked up by Nature, this was the first time that AI agents were better than humans at a real-time control task, not turn-taking…

Full transcript with timestamps

PETER STONE: Thanks, Dawn, for inviting me. and thanks to all of you for being here. It's a pleasure to be joining you. So what I'm going to do in this session in this talk is, as advertised by the title, I'm going to talk about my view of autonomous agents, and in particular a broad view, not just the latest incarnation of, quote, "agentic artificial intelligence or agentic large language models," but really the roots of the term autonomous agents with emphasis on embodiment, interaction, and learning. And what I mean by embodiment is really robots.

I know I looked through the topics that have been covered so far this semester, and there's been a bunch on learning and a bunch on interaction. So you had from Noam Brown, you heard about multi-agent systems. And also I think your last session was on multi-agent systems and also a lot on learning and scientific discovery. But I think there's been less coverage so far of the embodiment side, which sometimes is physical robots, sometimes simulated robots. And so I'm going to place an emphasis there. I'm going to start by giving a broad overview of where I'm coming from, the kind of research I do in my lab, and so tell you about a lot of things very briefly, so almost nothing about a whole bunch of things.

But then I'll do a deeper dive on one or two things, depending on how much time we have, that go into more of the technical details. And of course, there's more information on my web page and my papers about all the things I'm going to talk about here. And you're always welcome to reach out if you have questions about things that I only touch on briefly. So first, I'm going to just talk about what my view is. And I'm jumping between a bunch of different slide decks here. So apologies if it's not one unified view. Hopefully things will go smoothly enough.

But my view of agents, autonomous agents have been a focus of study in the field of artificial intelligence for a long time, for at least three decades, actually, more than that. I've been involved in the field for around three decades. And so much longer than this year's buzzword of agentic AI. But what is-- fundamentally, what is an agent? My view is, very simply, an agent is an intelligent system that interacts with the environment through sensors and effectors, as shown in the diagram here. I mean, you could ask the question, is a chatbot an agent?

Usually, we think of an autonomous agent as one where there's not necessarily an input-output sequence and then it sort of waits for more input, but rather more of a continuous loop where there's sensors coming in from the environment, or sensations coming in from the environment, actions that are taken that affect the next sensations but you don't have to wait to be called or invoked. And so there are software agents. There are also physical or embodied agents, which usually go by the term robot. The first slide was a modified slide from a definition in a term when I-- or a class when I teach robotics in the first session, I usually ask what is a robot.

And it's almost the same definition. It's a robot is an intelligent system that interacts with the physical environment through sensors and effectors. So the only word I've added here is the word physical. But otherwise, so all robots are agents, not all agents are robots. And then we often think about what's an intelligent complete robot, or you could substitute the word agent here, an intelligent complete agent is one that has some cognition, the intelligence aspect. So it gets the sensations, it has the actions.

And there's the raw sensors and the raw actuators, which if it's a robot, you can think of as being the cameras and the motors. But then there's the intelligent part shown in green here, which is the sensing and modeling the world based on the raw sensor perceptions, sensor readings, that is the perception module. On the other side, the far other side, there's the action aspect, which is, once you've decided what you want to do, how do you actually do it? And then translate that into the motor commands, the low-level actuators.

And in between is what many people consider the brains of the system or the cognition where you're deciding what actions to take. And that could be through planning, through multi-agent coordination, teamwork, opponent modeling or teammate modeling, et cetera. So all of my research takes this view of what an agent is. And I just wanted to frame that in the context of this talk because I think-- like I said, nowadays, there's some people take a more narrow view of what an agent is or an agentic system is, but really it's quite a broad term and broadly encompassing.

So as I said, and I'm going to start by telling you almost nothing about a lot of things. And starting from where I'm coming from, I am here at UT Austin, where there's a lot of exciting things happening. We were awarded one of the first National Science Foundation AI institutes, the Institute for Foundations of Machine Learning. This is an organization that spawned the Machine Learning Lab that's headed by Adam Klivans and Sanjay Shakkottai. We also have an ethical AI system on campus that's inspired by the One Hundred Year Study of AI that I've been a leader in.

But the one on campus here at UT is called Good Systems, and it involves people from many different disciplines and the ethical AI initiative. I was the founding director of Texas Robotics. We have a beautiful new space. We're going to be hosting-- this is before we filled it with robots, it's an old gymnasium that's now filled with robots from across campus-- we're going to be hosting the conference on robot learning here in about a year next November. So I hope many of you will have a chance to come on campus and see that.

And there's also a lot of innovative AI education going on here. We have an online AI master's program at a price point of $10,000. We have created an AI literacy course for the entire campus with all of the materials online. So these are all things that I'd be happy to talk about in much more detail. But from the perspective of my own research in the things I'm going to talk to you about today, the unifying question that's held together, all of the different things that go on in my lab, and have over the past several decades, is this, it's what degree can autonomous, intelligent agents learn in the presence of teammates and/or adversaries in real-time dynamic domains?

And I'll emphasize, again, this question was the core question of my PhD thesis back in 1998 and has appeared in almost every talk I've given since then. And it has included this term agent. So that's a way of emphasizing that this is not a new term. It's not a new buzzword, but really something that's been around for a very long time in the community. This question causes us to focus in my group on publishing in several different subareas of AI autonomous agents, multi-agent systems, robotics, machine learning, especially reinforcement learning.

And we do both foundational bottom-up research and more use-inspired research. And when I give a one-slide introduction to things that go on in my lab, it's a lot easier to talk about those use-inspired or the use cases that motivate the research. So one has been that I focused on a lot and found inspiring is robot soccer. So here's some of the old Sony AIBO robots. This is a video from probably almost 20 years ago of the robots sensing, deciding, and acting no remote control from the RoboCup, or robot soccer competitions.

We've now moved towards humanoid robots. So this is a clip from also a competition several years ago, where we were playing a team from the University of Bremen. Our robots are the ones with their hands behind their backs and pink waistbands. And again, remember that they're fully autonomous. They're sensing, deciding, and acting like an agent does. This is a multi-agent system. So there's several robots cooperating together. Here, you see a robot going in on a breakaway, and it'll end up scoring a goal. And we went on to win this game and winning the competition that year.

And there's all kinds of agent challenges that come up in this kind of a setting. The goal of RoboCup, the Robot Soccer World Cup, is to create a team of humanoid robots that can beat the best World Cup champion on a real soccer field by the year 2050. And so this is an ambitious goal, but it's sort of in that line of AI systems that have done better than the best humans at various tasks, from chess to go, to poker. You heard about, I think, about StarCraft in this course already. And I'll talk about another one as we go, but soccer would be taking it to the next level in many ways.

And if you watch that video, you might be saying, oh, well, that's not even close to human level. The robots are still not close to human level, but there's been big leaps. So that's roughly what it looked like at the competition two years ago. But in this year's competition, we had a game of humanoid robots for the first time with people. And you'll see me here with the ball. And I was playing on the team with the robots, because it's still too easy for the people to beat the robot. So it was three people against me and the robots.

And I had no idea what was going to happen. The robots didn't actually even sense me, but I left the ball for this robot here that I'm showing with my pointer, and you'll see what happened. And this was very unexpected, but shows you the level at which we're getting to now with robot soccer. So the robot just turned around and kicked it between the RoboCup president's legs, that's Ubbo Visser. I tried to get the robot to give me a hi-five afterwards, it didn't. But in any case, we're at this level where-- and we would never have been able to see that, I think, even a couple years ago, or even a couple months ago, let me say.

And so there is a lot of progress. And I think it's not completely out of the other question that by the year 2050, we would have humanoid robots playing at human professional level. I wouldn't bet my life on it. I think they're trying to make systems like that happen in the real world, in physical systems is much harder than doing it all in software. But there's been a lot of progress in the past 25 years or so. I'm also inspired by general-purpose service robots. If you come to my lab, you'll often see robots like this one just sort of wandering through the hallways in this kind of a setting.

And actually, the RoboCup competitions involve general-purpose service robots as well, doing things like acting as a host, having to pick up items to set the table, put away groceries. Here's the task serving breakfast. So there's a competition called RoboCup@Home, trying to create robots that are generally capable in this way, and that's been very inspiring to my research as well. And then, like many, I've been inspired by autonomous driving. An autonomous car is an agent. It has to sense, decide, and act. I had a car in the DARPA Urban Challenge, the one pictured here.

I don't have that car anymore. As you know the car companies and companies like Waymo and Zoox are putting a lot of money into making this a reality. But we've thought over the course of several years, and actually a couple decades now, what the world would look like when all the cars on the road are autonomous? Would we still need traffic signals and stop signs, or will we be able to see systems more like this? This was the basis for Dresdner's thesis back in, might have been 2010 or so. But this video, I think, came out first in 2004.

And what's happening is that each car is an agent calling ahead for a reservation through the intersection. And once it turns white, it has a guaranteed path through the intersection that won't interfere with any of the other cars. When they're yellow, they don't have a reservation yet, so they often have to stop. But this is when all the cars are autonomous. We may be able to see intersections like that. We've also looked at what happens if you have human drivers mixed with autonomous drivers and things like that.

And so I could give a whole talk on any of these topics. Most recently, I mentioned that I've been inspired by these challenges of trying to get AI agents to be better than humans at various tasks. And so with my group at Sony AI, we actually, a couple of years ago, contributed to that stream of results. We had a paper on the cover of Nature, where we created reinforcement learning agents that were able to drive faster than the very best human drivers at the game, Gran Turismo, which is a very physically realistic simulation environment.

So this was the first time-- the reason it was picked up by Nature, this was the first time that AI agents were better than humans at a real-time control task, not turn-taking like poker and chess and go and things like that, or I should say a physically realistic real-time control task. And the one way to get a sense of that is to watch above and below, an expert human driver, Ken Chan, driving in the simulator in the top half of the screen and in the real car on the bottom half of the screen, where he's controlling the same car on the same track in the simulator versus in the real world.

And it models sufficiently the slip stream and tire slip and traction and things like that, so that he was able to finish the driving at his best in both the real world and the simulator. He finished the lap within a 10th of a second of the times in the two modalities. And there are people who have trained in this simulator to become professional race car drivers. And so we did a race against four of the best human drivers. This was in October of 20-- we first raced them in July of 2021 and did not win-- the humans were still better than the AI agents.

But then by October of 2021, there was an event that's the basis for the Nature article from February of 2022, where we beat the four of the best drivers in the world who've put in their 10,000 hours to become real experts on three different race car track combinations. And since then, it's actually been deployed in the video game. If you have a PlayStation and the game Gran Turismo, you can race against GT Sophy. There's been a lot of research beyond the paper to make it more engaging and fun. But fundamentally, this was a success story for end-to-end reinforcement learning, where the agent started acting randomly, literally just driving in circles or backwards or off the track, crashing into things and getting reward for moving fast around the track.

And after about a day of training on a bank of PlayStations, it was able to do better than the best humans on an empty track, and after about a week, better than the best humans in a competitive racing scenario, where it learns skills like this one, where the colored car is the agent GT Sophy, the white ones are all human drivers. And here you see it learning to take a defensive line, so it's not going the fastest it can around the track, but rather positioning itself in such a way that it's hard for the humans to pass it.

And it's learned to do that entirely on its own. So this is work from a large team of people at Sony. And like I said, it's ongoing, and it's something that we're really proud of. Most recently, with a different team at Sony AI, in fact, very recently, this is just last week, or maybe within the last two weeks that this was published in Nature, we contributed a new human-centric image benchmark for-- basically, as I said here in the tagline, and this was led by Alice Xiang It's the first globally diverse, consensually collected fairness evaluation data set for a wide variety of human-centric computer vision tasks.

So really what sets it apart, our goals were to identify the ethical and legal issues of creating data sets, rather than just scraping YouTube and Instagram and Reddit and things that go into training these large models. Our goal was to try to, at least at the scale that you could for evaluation, create a data set where all of the participants, everybody who appears in the pictures, has been compensated and has self-identified what the label should be that describe them. And so tried to start with FHIBE, tried to establish an industry standard for AI ethics.

And also, it's built ethically, but also to evaluate bias in AI models. And so in a way of creating, and I'm not going to read through the slide here, but you can see some of the features of FHIBE. And it comprises 10,000 images. So it's not enough to train a model on, but it's enough to evaluate a model for biases. It's the first one that goes across the whole global geography, 81 countries and areas, and categorizes a whole bunch of different features, as you can see here. And annotations of the camera that are used, the environment, and the features of the people in the image itself, as labeled by the person who appears in the image in many cases.

And so we use this in the Nature article, which you can find online now do analysis of the sorts of biases that happen in visual language models. And I verified some existing ones that were known and also identified some new ones. So that's, in a nutshell, the kinds of things that inspire me and the research that I do have. As I said, you have-- another theme that I think comes up a lot in agents, and you've heard a little bit about during this semester, is human-in-the-loop learning, which is learning from interaction.

And there's a system that was developed by a former student of mine, Brad Knox, who's now faculty here at UT Austin, that was a system called TAMER, Teaching an Agent Manually via Evaluative Reinforcement. And the idea here, it was the-- actually, I think, it has a claim to being the first system to do reinforcement learning from human feedback, or RLHF, which is a key point, a key aspect of training language models now. But his core demonstration was in the game of Tetris that you see here. Most of you are probably familiar-- you're trying to clear rows.

And what's happening in the top-left corner, a person is saying, good move or bad move-- green for good, and red for bad. And at first, it was random. This was a task that at the time of the system, you could do with reinforcement learning if you trained for hundreds of thousands of games. But Brad was able to show that even in the first game, with this human feedback, you can start doing much better than random. So it's already starting to clear some lines. Oops. And then by the third game, it's able to-- so this is after just two games of a person-- two episodes of a person providing positive and negative feedback.

It's already able to clear hundreds of lines. And so this was one of the early examples or demonstrations of the power of learning from human interaction, as opposed to just random exploration. GT Sophy was purely from random exploration. It was all reinforcement learning. This is reinforcement learning from human feedback. And one of the key lessons here was that TAMER learns more quickly than autonomous learning from straight reinforcement learning. It doesn't take hundreds of thousands of games. It takes three games to get to quite a good level, but ultimately doesn't do as well as if you allow for reinforcement learning to learn from scratch.

You may have heard of the bitter lesson from Rich Sutton that injecting human input could bring down the ultimate performance. But there are also ways of combining them. So we had an award-winning paper TAMER plus RL, which learned both from human feedback and autonomously. And this was also with Brad Knox, part of his thesis. Then collaborating with Garrett Warnell, pictured on the right, we had an extension to video or image input, a system called Deep Tamer. And like I said, this really was, I think, the first example of RLHF, which has really been super impactful lately.

And then this has been extended to also learning, not just from explicit feedback but learning from implicit feedback. So not just a person explicitly saying good move or bad move, but you can imagine if you were an autonomous car and your passengers looked like this, you wouldn't need somebody to tell you-- shouldn't need somebody to tell you that you've done something wrong. You've scared them. And so this should allow you to learn from implicit feedback. So we, a couple years ago, in a coral paper, introduced a system called EMPATHIC, for Evaluative MaPping for Effective Task-learning via Human Implicit Cues, but it's about learning from implicit human feedback.

And then we've gone on to think about all the different ways in, or many of the different ways, in which humans can provide input in machine learning through the robot navigation problem, where a robot has to start at a point and get to a global goal while avoiding obstacles. I'm not going to go into the details on this again, I'm just going to show you the punch line where we were-- but it's trying to learn to have the robots move through very constrained spaces, which is not a solved problem yet. So it's navigation, is pretty well known how to do it.

But if you're trying to navigate through very constrained spaces that are not much bigger than the robot, there are navigation packages out there-- oops, I wanted to go to the previous slide here. Yeah, there's navigation packages out there, but they require lots of opaque parameters, things like this, that you wouldn't really know how to tune, if you wanted the robot to work well. And they're different in very different settings, whether the robot's in a corridor or an open space or an obstacle course or a pinch.

And so right now, if you want to get a robot to navigate effectively in these kinds of spaces, it takes a roboticist to fine-tune those planning parameters, or at least it did. And so we explored various ways of using human input to allow for a robot to learn how to do this. So ranging from a teleoperated demonstration where a person is literally just remote-controlling the robot, and then we have the robot explore in parameter space to come up with the parameters that would lead to the most similar behavior to the slightly less demanding corrective information or interventions where a person doesn't need to always be remote-controlling the robot, but can just step in when the robot is doing something wrong.

And again, now the robot can try to emulate or predict when a person would step in to intervene. We had a system for doing it from evaluative feedback, which is much closer to the TAMER system, where the person just has to say good move or bad move, give positive and negative feedback, all the way up to reinforcement learning systems. And so you can also find a full talk on my website about the various different technical algorithms behind doing this Adaptive Planner Parameter Learning, where rather than learning all of the different parameters of a navigation stack, we're instead treating that navigation stack like a black box and learning just the parameters that will lead to the behavior that you want.

And so the way I think of this is in the language of machine learning. We're learning a parameter policy. Rather than outputting the raw actions, we're outputting the parameters to a navigation stack or a classical motion planner, and trying to learn which ones lead to the actions that you most desire.

I think that's where I'll stop in terms of telling you almost nothing about a lot of different things. Like I said, when I looked through the topics that you've covered so far, there have been, I think, less on the physical robots than on the multi-agent systems and the reinforcement learning that I focused on a lot. Oh, actually, sorry, I take that back. There was one more thing I wanted to tell you almost nothing about, which is, on the interaction side, a problem that's fascinated me over the past, decade-and-a-half or so, called ad hoc teamwork, which is a particular type of interaction, where the goal is to try to create an individual agent that is good at cooperating with teammates, but you don't necessarily get to, or you don't at all, get to control who those teammates are.

The kind of thing that people can really do well when they're playing pick-up soccer or pick-up basketball, but now we want robots to be able to do if there were a disaster-rescue scenario and lots of different people brought the robots together, we don't want to have to reprogram them to work as a team. We want them to be able to figure out on-the-fly how to be able to work together. And so there's a bunch of work in my lab also on this problem of creating a good team player, which is, I think, a part of the multi-agent systems challenge or the interaction challenge.

And there's a challenge paper from AAAI 2010 that introduces ad hoc teamwork as a problem. And then there's been a bunch of theoretical research and experimental research on the topic. OK, now, I really do-- that's the whirlwind part of this talk, where I've told you very little about a lot of work that pertains to machine learning, especially reinforcement learning and interaction especially ad hoc teamwork, multi-agent systems. But you've seen some coverage of those in the semester already. With regards to embodiment, I'm going to take a little bit of a deeper dive and show you a paper from just-- it was presented at CoRL just about a month ago.

And let me put this into full-screen mode. Why is this not working? Try it again. And this is the goal of-- I'm sorry, this should be coming up. It's the goal of trying to do reinforcement learning-- so there we go--

in the real world. So the motivation here is, and I should say this is the PhD thesis work of Jiaheng Hu, shown here, and he's co-advised by myself and Roberto Martín-Martin. And this paper was published, like I said at CoRL, the Conference on Robot Learning recently. And so the paper is SLAC, for Simulation-Pretrained Latent Action Space for Whole Body Real-World RL. And so the key here is trying to do real-world RL. I want to get rid of that little thing there. No, that's not what I want to do. I'm sorry. This is not going exactly how I was hoping.

Let's just keep it in this mode. So it's learning visuo-motor, contact-rich, whole-body mobile manipulation with real-world reinforcement learning without a single demonstration. So when you're doing learning in robotics, you can't do what we did with GT Sophy, which was learn on a bank of PlayStations, hundreds of thousands or millions of interactions. While we were all sleeping, the computers would be working. When you're working on a real robot, you have to work very, very efficiently with very little data. And so you have to learn very efficiently.

And a motivating task here is we want to do whole-body mobile manipulation. So this is a task such as wiping the blue off of the whiteboard while avoiding the obstacle of this trash band. And so it requires the robot to know how to move to avoid obstacles and not just to-- the standard way to do this with reinforcement learning would be to just explore lots and lots of random actions first. And you can do that if you have a simulator. But it's costly to build high-fidelity simulators. It's sometimes infeasible to build a simulator that simulates your robot well enough that if you just learn in simulation and apply the policy in the real world, it would just work.

There have been some successes of that, it's not always straightforward. And so, we want to try to learn without the benefit of the learning happening in the simulator. One way people often get around that is through imitation learning or some kind of learning from human feedback, as I showed with the adaptive planner parameter learning. But that can be very costly to collect and the high quality demonstrations. So our goal is to do real-world reinforcement learning with directly learning from real interactions without this high-fidelity simulation or demonstrations.

And so, like I said, if you just use a standard state-of-the-art reinforcement learning method, you're going to just flail. So if you're trying to learn to wipe the blue mark and you're getting reward for wiping the blue mark, and you're just sending random commands to all the motors of the robot moving the base around, randomly moving the arm around randomly, you're not going to get very far. It's going to probably the battery's going to die, or you're going to lose patience before it learns to do anything correctly.

And so there's a huge challenge for trying to learn with sufficient sample efficiency and safety in the real world. And so the system that Jiaheng came up with is, like I said, it's called SLAC. And the key here is that we need a good action space to facilitate sample efficiency and safety in the real world. If we have, rather than sending raw commands to the robot, but have a much more abstract action space, then there's much less to explore, and we're going to use a low-fidelity simulator to come up with that latent action space.

So we're going to assume that it's not that hard to come up with a rough simulator of the robot, and that we can then use that to-- so here's a picture of what a low-fidelity simulator might look like. So it's sort of models the robot, but it's not going to be perfect. And we want to use what's known as unsupervised reinforcement learning to learn task-agnostic behaviors in a low-fidelity simulator. So what do I mean by unsupervised reinforcement learning? That means giving a robot or an agent an environment to explore in, but no reward signal, no task.

So it's unsupervised because there's no reward signal, and get it to try to just figure out all of the actions that it might want to take, that might be useful when it is given a reward function downstream in the future. So the kind of thing where we put a baby in a playpen-- a human baby in a playpen, they tend to try stacking blocks up or squeezing things to see what noise they make. They don't really have any task to do, but they're, in some sense, training themselves to do things that might be useful once they are given tasks.

And so that's the idea, is trying to get a robot to do the same sort of thing. That's what the unsupervised reinforcement learning idea is. And the goal here is to come up with the types of actions that you might want a robot to do. And so what we're trying to learn is a latent action space where you give just-- you define a number of dimensions, some small set of dimensions, and then try to make it so that when you change the latent actions, you get meaningfully different behaviors in the real world so that this latent action space can create-- you might imagine that the robot would learn to get close to the whiteboard versus getting far away from the whiteboard.

Those are something that you can do by just moving the base of the robot, or you could imagine another action dimension controlling the arm or where the camera is pointing. And so we came up with a method called DUSD, or disentangled unsupervised skill discovery, and this was before the SLAC paper, that does learn this sort of diverse, temporally extended behaviors by trying to come up with-- by maximizing or minimizing the mutual information between the behaviors you get when you change the latent action features.

There's been previous work. There was a paper called Diversity is All You Need, or DIAYN, from Sergey Levine's group that had that insight initially. But one of the things we added is that we tried to do it in a disentangled way. So to create a structured action space, by which we mean that if you vary one latent variable, it only changes one dimension of the action space and also adds safety. So what I mean by that is we want to add two different reward signals. One is such that if you change one of the latent action features, it only changes one aspect of the robot's behavior.

So here the red indicates where the camera is pointing and the blue indicates where the arm is moving and the red indicates where the bass is moving. We wanted sort of a nice structured action space so that you can-- if you vary one of the latent action features and hold the others constant, it'll only move the base, the arm of the camera. And so that was something that we were able to do using DUSD, using what we call the skill empowerment reward. And then we also added a hand-coded safety reward that's like don't collide with yourself, something like that.

And when you put those together, you can come up with a-- train the action decoder in this and the latent actions such that you get this structured and safe set of actions in the low-fidelity simulator. And then once you have that in the second step of SLAC, you can then take the learned latent action space and use it for downstream task learning in the real world. And so, again, you have this lower dimensional action space where each dimension changes the different aspect of the robot. And so then we're going to separately develop a factorized version of Soft Actor-Critic.

Soft Actor-Critic is one of the popular reinforcement learning algorithms. It's actually the basis for the one that we used in GT Sophy as well. And it leverages the structure of the latent action space. And so what I mean by that is that once you're given a reward, once you know what your reward terms are, you might have a reward for where the robot is located that can be controlled by where the base is, a reward for where the robot is looking that can be controlled by where the camera is pointing, a reward for what you're picking up or touching, which is controlled by where the arm is.

The idea is that not every latent factor controls every reward term. So you can get a much sparser and more accurate prediction of the value function in reinforcement learning terms, if you can learn to separate which action factors control which reward terms. And so we did this in a paper that appeared at RSS 2023. We used a conditional mutual information metric that allows us to say, well, if I change a particular action dimension, how does it change the reward factors that I get. And so you can learn a masking matrix that says, if you want to-- if you have a reward term that only depends on where the base of the robot is, which action factors do you need to vary, or can you vary to get that reward, and so on for every other factor.

I know I'm going quickly here from a technical perspective. I'm not going through all of the math, but you can find this in the papers that are referenced on the slide. And so then we can use all the data that's used to infer which action factors affect which reward terms for learning. And I'll get to the punchline and the video demonstrations here coming up. We can take some real-world tasks and then do learning entirely from the onboard observations from the robot's camera and proprioception, which is where the joint angles are at any given time.

And now learn a task policy, the one that actually gets rewards for the task, like in this case, it's sweeping something into a basket with your frozen pre-trained latent action model, which is what you use the low-fidelity simulation for it to come up with this structured action space. And then if the action space is low-dimensional enough, you can actually learn to control the various reward factors, where to look, where to move your hand, where to move the base by exploiting that causal matrix from the previous slide that allows you to change the various action factors to get the various-- to maximize the various reward terms.

And then here's a demonstration. So at the start of training, a robot has no idea how to solve its task, just like in the previous video I showed you. So it's sort of moving around randomly. So here's the first minute or so of training minute or two. But after only 40 minutes, so it gets a behavior now where we've given it reward for being able to successfully wipe off the blue marks from the board. And it's learned to successfully move the base, the camera, and the arm in unison to be able to achieve that task. And again, with all of the starting from random behavior and 40 minutes later having learned this skill.

But by leveraging this very efficient action space that was learned in simulation. Using exactly the same methodology, we can end that same latent action space. It can learn to sweep garbage onto a tray or garbage into a bag. And also it can learn to wipe the whiteboard with or without an obstacle in front of it. And so we've done these four different tasks that I just mentioned. And compared to doing all of the learning in simulation or a couple of existing methods, one that which learns entirely in the low-level action space rather than in this more factorized abstract action space.

Also another method called RLPD. And we both have higher success rate at those four tasks, and importantly, fewer safety violations during learning so that the robot can actually do this learning efficiently in the real world. And in all cases, all of those policies trained in less than one hour on the real robots so that we end up with an autonomous agent that's able to learn in that case. So you can learn a lot more details from the paper. And this is a theme for a lot of the recent research in my lab is trying to use simulation, but to allow for learning in the real world, but not necessarily by just learning all in simulation and transferring the policy directly, but instead leveraging the simulation in more creative ways like here to pre-train an efficient action space.

And of course, like many, we've also been exploring how can we use foundation models, large language models and visual language models to not necessarily control the low-level actions of the robot, but to provide some of the common sense knowledge for the robot. So that's what I wanted to do in terms of the embodiment, sort of a somewhat deep dive into a particular research paper on embodiment. I think I have time to also go in a little bit of a deep dive on GT Sophy as well. So I've given you an overview of this.

Oops, where is it? Oh, I think I have it here. There we go. So I've given you an introduction to GT Sophy, but I'm going to tell you a little bit more about it now and then I'll pause for questions. So again, to remind you, this is the work that was done at Sony AI on racing champion Gran Turismo drivers with deep reinforcement learning. So this is going to that other extreme where we now do have a simulation. And actually the whole task is to do well in the simulation. So this is not embodiment anymore. This is to build a real world-class world-expert agent at race car driving.

And I showed you this slide already. But the success was just like in the success cases of chess and poker and Go, it relied on a great compute architecture. There was a lot of data that went in, a lot of engineering. It wasn't just sort of a new out-of-the-blue principle. So there's a lot of data and top-notch engineering and compute. But there also were a few algorithmic and methodological insights that went into making it possible. And I'll try to give you a sense of those right now. Those are what are featured in the article.

Just as a reminder, in every video, I show you, the colored cars are going to be GT Sophy and the white ones are controlled by one of the very best drivers at GT Sophy. I've already told you about the simulator itself and that we do this with reinforcement learning. The main challenges that made this more difficult than previous tasks was that it's real-time control. So we had to come up with actions at 10 times per second, where it's acquiring a deep understanding of the car dynamics, "understanding" in quotes, but the ability to control a car following racing lines, precision racing maneuvers.

And when I say at the edge, it means, if the car has-- the controller has to figure out if it goes-- how fast it can go. If it goes any faster, it's going to spin out. But if it goes any slower, it's going to be easy for a human to pass. It's got to be right at that edge of staying in control. There's also tactics, like I showed you already the example of defensive moves, but there's many more that I'll show you about. And also there's this really interesting and ill-defined aspect of etiquette, where you can't just push another car off the track.

There's a human judge that decides what's sportsmanlike and not, and the agent has to have that encoded as well. So in this case, the actions that the car used were at 10 Hertz. It could either control the brake or the throttle to be full backwards or full forwards, can turn full left or full right. It's really already a factored low-dimensional action space. There's just two dimensions to it. So we don't have to go through what we did with SLAC to try to discover that. The agent already has this natural factored, low-dimensional action space.

The rewards initially are just how fast it moves along the track. And then we added that's mainly the positive rewards and then also penalties for the tire slipping or going off the track or hitting a wall. And then there were initially in the initial work, we had not visual features. We have a couple of papers since then where we have used visual inputs to help the car control. But in the initial paper, we were giving things like the car's velocity, acceleration, roll, pitch, and yaw load on the tires and position along the track to help it.

The compute infrastructure was really important. But by doing this within the Sony ecosystem, we were able to partner with Polyphony Digital and Sony Interactive Entertainment for the PlayStation group to have access to a large bank of PlayStations, which we could then use in a sort of an Actor-Critic kind of scenario where we have a lot of roll-up workers getting data on what happens when you take an action and what state comes back. And then we used-- and we store that data in experience replay buffer and then use an Actor-Critic method based on Soft Actor-Critic, but introduced a new algorithm called quantile-regression, or QR-SAC, Quantile-Regression Soft Actor-Critic, to be able to train the policy, which is pi, that's what actions you take, and the critic, which is the value function, which is basically an estimate of how good it is to be where you currently are in the state that you're in.

And so here's a slide of technical details for people who are familiar with reinforcement learning. Don't worry if this doesn't all speak to you. But basically, QR-SAC was learning, not just the average or the mean value of taking an action, but the full distribution. What's the likelihood of a really bad thing happening versus a really good thing happening when you take an action. So it's learning a quantile or a distributional RL value function as it's called, and was also using N-step returns rather than one-step returns to improve its performance.

The numbers here on the right are the lap time in a particular configuration on a particular track, and so smaller is better if you can get around the track in 114 seconds. That's better than 115 seconds. So it's about almost one second improvement, which is pretty huge in this domain between the vanilla Soft Actor-Critic and our new algorithm. And we also found the sweet spot for how the N-step returns. The details are in the paper, and I have talks that go into much more detail on how QR-SAC works and the neural net training details.

But the bottom line is that after an hour, the car is able to move forwards, not run into walls or drive off the track, but not very efficiently. After four hours, to move much more smoothly, after eight hours, pretty expertly. And actually after a day, although we can run it for a week, it's really at superhuman performance, like a person can't control. And it's hard to see, I know, on online in Zoom, but it's cutting the corners just exactly right in the optimal way and in a way that a person can't do any better.

And in fact, it does better than almost every person in the world. We tracked it against after 4 hours, 8 hours, and 24 hours. The blue here is a histogram of the best 100-lap times reported by people on a worldwide server. And after 24 hours, it's better than almost all of the reported lap times. And if you take the average of a person, the best person's average is shown here on the left and the distribution of GT Sophy's times, better than the second, third, fourth, and fifth and on par with the best person in the world.

And that's just on this one track. On the other two tracks, that's significantly better than the best person's average time. And so that was the first step. But in this domain, being faster wasn't enough. There were two other challenges to solve. Like I said, there's the race car control, which I showed you we could do, but there's also the tactics and the etiquette. So to be able to do-- to bring that, we augmented the state space to include knowledge of where the other cars are nearby, their relative position, velocity, and acceleration for the cars in front and beyond and behind.

Also a reward for passing, a penalty for being passed that encourages cars to be a little more risky when they're near other cars to get around them. And then importantly, we had to curate the data. This was another one of innovations so that the car gets the experience that it needs. So we put a bunch of scenarios where it's just racing on the empty track. We can put 20 cars-- the important thing to know is you can put 20 cars on the track at any one time on a place in the game, and so we could put just 20 cars, spread out around to get a lot of data.

But then we also created some situations where there's one car behind one other car, or one behind two or one behind three or one behind seven, so it gets experience driving in traffic. And in particular, it gets experience where it starts behind another car on a straightaway where it can-- it's encouraged to but not explicitly told to learn what's known as a slipstream pass, where it just sort of like bicycle racing. You draft behind the other car so that you have lower air resistance, and you can get a lot of momentum and then bounce out from behind them and be able to pass them.

To be able to learn that, it's important that the simulator actually models this slipstream effect and it does. And then also, where you start behind another car in a corner, so you learn these sort of crossover pass maneuvers, and also a bunch of other tactics, like what happens when you're in a grid of the start of the car, or if you've got this particularly difficult parts of the track. And so one of the things that was a key to the success here was creating a carefully engineered mix of data in the experience replay buffer so that various points along the track, it would be learning from data from different places, and then create a mini batch for training that gives the car the right mix of experience.

And for the Nature paper, we did this manually, but then since then, we have worked on automated methods for coming up with the right mix of data in this experience replay buffer. So some of the skills that were learned by this agent were things like that slipstream pass where-- and these are narrated. You can find some talks online where you hear the sound as well. I don't have that working right now, but where the GT Sophy is the colored car again. It starts behind the other car, gets in that low air resistance mode, or your location and is able to pass.

So that's an example of that happening. I already showed you the defensive skill. And here's a really that double crossover, where you're going to see the two different cars-- the green and the gray, I think, start out behind the two human drivers. But by the end of this turn, they come out, both of them in front of the human drivers. And so you see on the right the position or the line that they took to be able to successfully do that crossover pass. And again, this was not programmed in or explicitly rewarded in any way.

It was just part of the overall learning process and the experience that we gave them. And then finally, there was this aspect of etiquette. I'm not going to go into the details either here, but there's things like trying to avoid avoidable collisions, forcing a driver off the track, doing things illegitimately that are all sort of judgment words. Things that a human Steward would have to assess whether they're happening or not. But if you don't do this, you'll end up with aggressive agents like the one shown here.

Here's also showing the interface of a person controlling the car. But here's what you're not allowed to do. You can't just push a car off the track like that. But you also don't want to be too timid. So this was a clip where GT Sophy was in front and the human announcer was saying, there's no way that the person's going to be able to pass. It's a commanding lead. But then GT Sophy actually hit the brakes because it was too timid, it was too concerned about causing a collision, and allowed the person to pass. And so we needed to come up with ways of mitigating that.

I think the human pass is coming up right around here. Right now, GT Sophy still in the front, but I think you'll see-- yeah, here's Tomoaki Yamanaka, the human expert, now he has finally passed GT Sophy because of that timidity. So we added a bunch of reward signals to make sure the agent wouldn't be too-- wouldn't get into collisions and also especially wouldn't cause collisions that it would be judged as being at fault for and penalized for. And so we did an analysis in the paper of how the different mix of opponents and the reward functions would lead to different aspects or different agents.

I'm running out of time, so I'm not going-- I want to leave time for plenty of time for questions. So there's also an important aspect here of, we trained lots of different agents, but how do we choose the one that's going to go into the actual race. That's an interesting aspect that's covered in the paper. But the bottom line is that in July, the humans outraced the people and so we had a rematch. And we analyzed what went wrong. We spent a few months improving it. And by October created a set of agents that was able to finish first and second place in an eight-car race for GT Sophy cars against four humans in a setting where you got points based on 10 points for finishing first, eight for second, and then I think 7, 6 5, 4, 3 for the rest, or something like that.

That was on the previous slide, I think, what the scoring was. Yeah, 10, 8, and 6 for the first three places and then 5, 4, 3, 2, 1 for the other places, and double points for the harder start task. But the bottom line is by October, we were able to finish first and second in all the races. And interestingly, there were some quotes from some of the drivers, including Emily Jones. I'm not going to play-- that's her there. But saying things like, I've learned something from the agents. I watched them, they're driving differently than I would and they've improved.

There are things that I ought to try. And now, like I said earlier, this has been permanently deployed in the game in GT Sophy, and there's been a lot of testimonials from gamers that this is a real big addition to Gran Turismo, and now it's out there on many more cars and tracks. But there's still a lot to do. One of the things that we're trying to get the agent to drive with style. So here's if we added a reward for drifting the car, learned to do something like this to go around turns, which is not something any human driver would do, but it sort of does a little spin-move, moving into a turn and optimally comes out of it, and also trying to make it more fun.

That was an important part of putting it into the game. It wouldn't be fun if you could just race against an agent that can outrace all of the best people. At least if I were playing the game, I would see the agent just for the first couple of seconds, and then it would be gone ahead of me for the rest. So we needed to have a way to make it also adjustable in terms of its skill level. So again, this was work of a group of people at Sony AI. And so I encourage you, there are hour long talks about this on my web page.

You can get the paper as well. But that's mainly what I wanted to show you today. I know it's been a little bit of a whirlwind. But I wanted to-- one of the main things I wanted to drive home is that autonomous agents research has been around for a long time. Agents are quite a general thing in intelligent systems that interact with the environment through sensors. Sorry, this is getting gotten jumpy-- there we go-- through sensors and effectors. And then an intelligent, complete agent is one that has perception, cognition, and action from those sensors in between the sensors and the actuators.

And that key components of autonomous agents, from my perspective, are interaction and learning, for sure, as I think have been big themes throughout your semester, but also embodiment, trying to get things working on real robots. And so with that, I invite you to, again, learn many more details about all of the things I mentioned today from my web page, from publications. If you look at my teaching page, there's a lot more online talks that have, in some cases, full one hour talks on things I just mentioned for a minute here.

But with that, I'll thank you for your attention, and be happy to take.

Questions about this episode

Who speaks in "Agentic AI MOOC | UC Berkeley CS294-196 Fall 2025 | Autonomous Agents by Peter Stone"?

Peter Stone (Main speaker), Noam Brown (Guest).

What is discussed in "Agentic AI MOOC | UC Berkeley CS294-196 Fall 2025 | Autonomous Agents by Peter Stone"?

In order: 0:00 Peter Stone, RoboCup, Adam Klivans, Noam Brown; 10:00 Nature, GT Sophy, Sony AI, Gran Turismo; 20:00 Brad Knox, CoRL, GT Sophy, Roberto Martín-Martin; 30:00 GT Sophy; 40:00 GT Sophy, Sony AI, Gran Turismo, Polyphony Digital; 50:00 GT Sophy, Gran Turismo, Nature; 1:00:00 Sony AI.

What is mentioned in this Berkeley RDI Center on Decentralization & AI episode?

24 people, works and ideas are mentioned, including Peter Stone, GT Sophy, Sony AI, Gran Turismo, Brad Knox, RoboCup. Each one on this page links to its own page with every episode that mentions it.

How long is "Agentic AI MOOC | UC Berkeley CS294-196 Fall 2025 | Autonomous Agents by Peter Stone"?

The episode runs 1 h 2 min; the transcript is about 9,755 words in 92 timestamped sections.

More from Berkeley RDI Center on Decentralization & AI

About this transcript: the episode's spoken words as captioned on YouTube, split into timestamped sections. Reclip.Me locates every person, work and idea in the text, keeps the time it is first named, and quotes the words around it verbatim. Last updated January 24, 2026.