Transcript

John Carmack (the creator of Doom) Reveals AGI Future: Robots, Videogames and AI Agents...

From Wes and Dylan · Episode page · Watch on YouTube

· 51 min · 9,523 words · 62 timestamped sections

This is the full, timestamped transcript of "John Carmack (the creator of Doom) Reveals AGI Future: Robots, Videogames and AI Agents...", an episode of Wes and Dylan published July 4, 2025 (51 min, about 9,523 words). It mentions 39 people, works and ideas, including OpenAI, Rainbow, Keen Technologies, ID Software, Nvidia.

What this episode is about

In this episode, John Carmack, creator of Doom and former Oculus CTO, pulls back the curtain on Keen Technologies, his secretive AGI research venture. He traces his path from pioneering first-person shooters and GPU adoption to VR systems engineering, and explains why he left Meta to pursue artificial general intelligence. Carmack makes the case that large language models, despite their impressive capabilities, cannot be the whole answer to AGI, and he shares his alignment with Richard Sutton's research vision.

Key takeaways

  • Carmack argues LLMs can't be the whole AGI answer, since human learning works nothing like training on the internet's blended text.
  • Original Quake's GL variant helped drive early GPU adoption, quietly laying groundwork for today's AI boom.
  • Carmack went from 'Victorian gentleman scientist' to founding a venture-backed lab, proving even solo geniuses need a team.
  • Keen Technologies is deliberately heads-down: no product, no fundraising, just unglamorous research on problems with no known answers.
  • Carmack's shift from engineering to science: he now chases knowledge nobody has, not endpoints he can already see.

What is discussed, and when

  1. – 5:00: Alberta Plan, Armadillo Aerospace, ID Software, Keen Technologies
  2. – 10:00: Sega Master System, Visual Studio Code
  3. – 30:00: Atari 2600, Konami Code, Stella
  4. – 35:00: Rainbow
  5. – 40:00: OpenAI, Rainbow, Sonic the Hedgehog
  6. – 45:00: Pitfall, Wolfenstein

Notable quotes from "John Carmack (the creator of Doom) Reveals AGI Future: Robots, Videogames and AI Agents..."

  • The other talks here. and now I know there's a little bit of an impression that Keen Technologies has been a little bit secretive since it's been announced a couple years ago.…
    about Keen Technologies · at
  • Of the prime driver for the early adoption of GPUs. And a couple books on Nvidia recently have kind of called that point out pretty specifically, and I feel quite good about…
    about Nvidia · at
  • But I did wind up parting ways with Meta eventually largely because while I was there, the OpenAI founders, Sam, Greg, and Ilia tried to recruit me.
    about OpenAI · at
  • Actually what I wanted to work on. Now, I had started off in sort of what I called it Victorian gentleman scientist mode. the idea that I'm a wealthy enough person. I can go do…
    about Victorian gentleman scientist · at
  • That he actually wants to work with me on this project and there's a lot of strong alignment between the Alberta plan and the way I think about things.
    about Alberta Plan · at
  • Working in notebooks before eventually migrating towards spinning up cloud instances using Visual Studio Code on remote systems and kind of working there. Another thing was…
    about Visual Studio Code · at
  • To go ahead and push over to the right side. And another example I mention a lot on that is like the old Konami code for old-timers when you remember you're going up down left…
    about Konami Code · at
  • Nothing particularly special here, it's it learns much faster than Rainbow, but not as fast as BBF. But add a few frames of latency to BBF and then it's well below that. And I…
    about Rainbow · at

Who and what is mentioned in this episode?

People

  • Richard Sutton — Person · 1 mention · first at
    Great journey for me. One of the real kind of surprises was joining up with Richard Sutton where part of my doing the reading and the research phase of I call the laral phase…
  • Greg Brockman — Person · 1 mention
  • Ilia Sutskever — Person · 1 mention
  • Sam Altman — Person · 1 mention

Companies & orgs

  • OpenAI — Company · 2 mentions · first at
  • Keen Technologies — Company · 1 mention · first at
  • ID Software — Company · 1 mention · first at
    So, backgroundwise, I started out in the early 90s founding ID Software, where we made a lot of foundational video games, kind of defined the first-person shooter genre.
  • Nvidia — Company · 1 mention · first at
  • Armadillo Aerospace — Company · 1 mention · first at
    There. kind of con the latter half of that. I also spent some of my time doing Armadillo Aerospace building vertical takeoff and landing rocket ships. Interestingly, it was…
  • SpaceX — Company · 1 mention · first at
    Vertical takeoff and landing rocket ships. Interestingly, it was founded basically the same year that SpaceX was to substantially less general success.
  • Oculus — Company · 1 mention · first at
    Laid down most of the current foundational technologies for the modern era of virtual reality for Oculus.
  • Atari — Company · 1 mention
  • Meta — Company · 1 mention

Products & tech

  • Visual Studio Code — Product · 1 mention · first at
  • Atari 2600 — Product · 1 mention · first at
    Like an interesting thing is we're using this modern system called an Atari 2600 plus and inside it's just got like a little Raspberry Pi running Stella. It is not like a rebuilding of the original Atari system. So just that system adds.
  • Stella — Product · 1 mention · first at
    This modern system called an Atari 2600 plus and inside it's just got like a little Raspberry Pi running Stella. It is not like a rebuilding of the original Atari system. So…
  • Sonic the Hedgehog — Product · 1 mention · first at
    Starting with this or I found OpenAI had done this challenge with got to learn fast which was a challenge using Sonic the Hedgehog game kind of a somewhat more modern game than…
  • Pitfall — Product · 1 mention · first at
    That everybody had the most problems with they called them the hard exploration games like Pitfall and Monzuma's Revenge and those were games where you might play for minutes…
  • Wolfenstein — Product · 1 mention · first at
    That didn't have a score. Up through there, all of our previous games, the Commander Kees and Wolfenstein, we were arcade people. So, we had lives and scores. you ran out of lives.
  • AprilTags — Product · 1 mention
  • Commander Keen — Product · 1 mention
  • CUDA — Product · 1 mention
  • DeepMind Lab — Product · 1 mention
  • Doom — Product · 1 mention
  • Montezuma's Revenge — Product · 1 mention
  • PyTorch — Product · 1 mention
  • Quake — Product · 1 mention

Concepts & topics

Full transcript with timestamps

All right. Hi everyone. I so I did complain that this talk is probably mistracked in the business of AI. Uh while we are venture funded, we're very much a research effort. So it will be a little more technical than maybe some of the other talks here. Uh and now I know there's a little bit of an impression that Keen Technologies has been a little bit secretive since it's been announced a couple years ago. And it's not really because we're intentionally being secretive. It's just we're not raising money. We're not selling a product. So, we've just been kind of heads down working, but I am happy to finally start talking about a little bit about some of our directions here. So, uh, and also I have managed to go 35 years in my career without making a slide deck. This is my very first one. So, apologies if it looks like a middle school project.

So, uh, backgroundwise, uh, I started out in the early 90s founding ID Software, where we made a lot of foundational video games, kind of defined the first-person shooter genre. But relevant here, uh, the original Quake or its GL Quake variant was kind of the prime driver for the early adoption of GPUs. And a couple books on Nvidia recently have kind of called that point out pretty specifically, and I feel quite good about that. It's kind of led to the the AI world that we're in now. And then a little bit later on, some of the games like Quake 3 Arena was used for the the Deep Mind Lab. Uh although sanitized of all the kind of action and violence for generic task work there, but another little AI tiein there. Uh kind of con uh the latter half of that. I also spent some of my time doing Armadillo Aerospace building vertical takeoff and landing rocket ships. Interestingly, it was founded basically the same year that SpaceX was

to substantially less general success. uh but there's some lessons in there that I enjoy talking about. You can grab me sometime after the talk if you want to hear more. But then after that uh I laid down most of the current foundational technologies for the modern era of virtual reality uh for Oculus. And this was a big systems engineering task around uh kind of minimizing latencies through everything and figuring out how to make things cost-effective enough to work on mobile systems. And the current Oculus Quest I'm very proud of. I'm proud of what we accomplished there.

But I did wind up parting ways with uh with Meta eventually largely because while I was there, the OpenAI founders, Sam, Greg, and Ilia tried to recruit me. And I was pretty flattered by that because I'm not an AI guy at the time. I was a video games maybe a perception and I'm kind of analysis sort of person but I had largely not paid that much attention to all the exciting things that had been happening in AI but once they kind of prodded me about that I sat down and I started doing the reading seeing what the state of things were possibly fit into it and it was really exciting. It was to the point where I looked at this and thought this is actually the most exciting thing anybody could be working on right now. And someone in my position uh deep technical background, the freedom to do whatever I want, I it was clear to me that this is actually what I wanted to work on. Now, I had started off um in sort of what I

called it Victorian gentleman scientist mode. you know, the idea that I'm a wealthy enough person. I can go do science by myself and research the different things. But I had a number of people that kept pushing me. It's like, no, you should spin up a company. We should get some resources to you. You should hire more people. And eventually I gave in on that. Um, and we did take some venture money. And we now have six researchers working at the company. And it's been um, you know, it's been a great journey for me. One of the real kind of surprises was joining up with Richard Sutton where part of my doing the reading and the research phase of I call the laral phase of getting into any new technology was reading his textbook you know twice and going through that and then finding out a year or two later that he actually wants to work with me on this project and there's a lot of strong alignment between the Alberta

plan and the way I think about things. um you know the the general hotness you know where we had LLMs being everything everybody like all the cool kids are doing an LLM startup and raising huge amounts of money to do things and I am you know I am amazed and excited about all the things that are happening there I'm a daily user of many of these technologies but I do still look at this fundamentally is that LLMs can't be the whole answer you know these uh transformer-based models it's not the way a human brain works and what they do as magical as it is is not handling so many of the fundamental things that you know cats and dogs do let alone small children where a lot of people don't you know they imagine that these LLMs are reading all of these books reading the whole internet and they don't understand that no it's taking all of human knowledge putting it in a giant blender and then training from there and it's

magical it works really well but when you're put into a situation where you have to learn something new. There are fundamental things that so many of the researchers here are working on that are just not understood yet. We do not have even a line of sight to an answer for these things. And this was the change for me where everything that I've done before has been some kind of engineering where even if I'm doing something newer or better than had been done before, I had the line of sight. I knew what the end point was and I had a pretty clear idea of what I had to do where this is doing science and this is trying to find knowledge that nobody has right now but it does feel like it's the time of all the happenings where the next few years is where things are really going to happen and so I had an obvious background in virtual environments and games so this idea of simulation and working with games I just kind of

naturally flowed from what I was doing and I think that uh that's still a very valid direction. There is this other side of touching reality and finding the things that are the limits of simulation that I you know that we'll get back to. But at the time I thought that video understanding was going to be one of the necessary steps to tie with all of these problems with game playing and reinforcement learning. Uh so I kind of imagined this uh infinite video wall that was interspersed with regular TV content and interactive games. And I was going to set out to have my agent learn how to play multiple games in a continuous environment with intrinsic rewards and all of these standard goals, but interspersed so that it also had the ability to learn from static video feeds, you know, like human beings do as infants.

As it turns out, I kind of think I made a few missteps that were somewhat predictable in some ways here where I started out as Mr. Lowle. you know, I'm the guy that writes assembly code or CUDA kernels and does all of this and that was kind of looked at as one of my core strengths. When I started doing AI research, my first several sets of programs were built from that standpoint. And there is great value in kind of learning the very nuts and bolts and knowing exactly how something is implemented and how something might be done differently. But in hindsight, I look back at as a mistake. I can look at how I migrated up from custom CUDA kernels to using coup blast to using coupnn to using libtor torch to eventually I'm in pietorch like everybody else and it's you know and it's a good thing I also probably avoided some largecale cloud utilization longer than I should where I again had a little bit of a bias towards physical

hardware you know I I like the idea of running something on a GPU that I could put my hand on and hear the fan worring on. I spent too much time kind of like working in notebooks before eventually migrating towards spinning up cloud instances using Visual Studio Code on remote systems and kind of working there. Another thing was just to sort of be contrarian. I had started off working with Sega Master System as my kind of gaming of choice which nobody had worked with and I thought well this will be interesting sort of untrod territory.

It's a little bit of a step up from Atari in terms of complexity, but not having any baselines of anything else to compare against was uh was a mistake. And then I also decided that the video understanding really can wait. That so many of these core problems that I look at as the unsolved science right now can be handled with just sets of games. you know, not necessarily a single game learning, but we can build a rich environment out of sets of different games and address these kind of challenging science problems. So, we did perhaps boringly decide to settle in with Atari and you know the Atari learning environment is uh you know it's born in Alberta and it underpins so much of what happened in dynamic reinforcement learning and there is a little bit of this sense that uh maybe it's played out now. is this not the where the hot research should be going on that there are other things that that

might be better but I do really think that it's got all the necessary things and some of the critical aspects are uh you know the diversity of having 100 plus different options there is valuable being unbiased by researchers is I think undervalued where anytime a researcher builds their own benchmark builds their own uh kind of environment to play in it is unavoidable able to have a bias to not do the things that you think are going to be hard for your algorithm. You've got your problem you're trying to solve, but there's always a whole host of other side effects for things. And if you build it yourself, you're going to build something that's not going to do horrible things to your algorithm.

That's just the nature of humanity. While the Atari games were built for humans to play, they were built before any of this research even existed. So they exist there as an unbiased sample of challenges for us to undertake and there are huge challenges where you might get spectacular performance playing one game make a change and it's better there but then it's worse in another one and this richness I think is uh is sufficient for a lot of these things that we want to do now the standard Atari games the way since DQN 12 years ago or whatever have been tested in this regime where you take one agent, you train it for 200 million frames, and that's a lot of play time.

It's like a solid month of play time, um, just 247 running a game like that. And as far as scores go, the very latest agents, things like Musli and meme, they are generating highly superhuman scores on almost all of the games. So it can look like it's solved like okay we can destroy these games better than any human can but there's a lot of other aspects to it where uh learning quickly is a good thing and I understand it's not a well-loved benchmark but the Atari 100K benchmark that people have been doing for the last few years I think is a really good way of addressing that where saying you've only got essentially two hours of game play to learn how to play a game that uh that makes it much more challenging and you could take these very great worldbeating agents and say you've got two hours and they don't look nearly as impressive. You know, all of these games start off looking terrible

and it takes a while for them to get up to the point where they're doing something interesting, let alone something superhuman. So, I think there's still a lot of good value here. Now the spectrum of choices for environments on the one hand you can start with something like grid worlds where you've got these synthetic academic environments where you test a very focused and specific thing but obviously that's really limited and on the other end you can have sophisticated real world interactions like I mean self-driving cars is is a crucable of all the problems of real world tied to reinforcement learning or not reinforcement learning exactly but machine learning in general But in between there's a spectrum of games where Atari was sort of the grandfather of all these games and they're simple generally scene-based games. Uh common question is well why not something modern something that people play today

like Pokemon or Minecraft. And uh I know it's the same argument that pushes people towards grid worlds but if you can solve a problem or if you can grapple with a problem in a simpler environment you probably should uh rather than having unnecessary complexity on top of it. So it's still kind of a judgment call there. But also when people undertake working in these modern games, the temptation is so great to look beyond the pixels and look at internal data structures. And honestly, I still just look at that as cheating.

You know, it's a way to get your your your agent to do something that looks interesting in a relatively reasonable amount of time, but you're bypassing most of the hard problems. if you're checking on inventories or positions or something, you're not worrying about maintaining your track through, you know, through a history of experience. So, I Atari is a long ways from being solved for all of these hard problems. One of the big points I want to push on the the broader community of people that are playing a bit with Atari in the research community is this that reality is not a turnbased game where the classic RL system right now is a simulation environment where you've got your agent, you call off to the environment and say, "Here's what I did.

Tell me what happened as a result of my action." And it pushes back a new observation and a reward to you. And that is very much like what a board game is like chess or go and and RL has done really spectacularly there. It's not like most real world interactions. And now it's true that things like the the cheetah and hopper and I those kind of continuous RL tasks do have many more of those um those kind of complications but they're also then tuned towards these are research toy problems and don't have some of the other richness there. Uh, so I I do push for this for the Atari work, anybody that's working on that to sort of invert the RL environment where think about the environment calls the agent.

You know, the agent just gets called. The environment probably keeps going, keeps on doing its business. If you take a really long time to come up with the next action, it will have gone and done many things after that. And and then if you're actually working in real real time, you have the issues of you just can't spend all of this time. Like some of the the really spectacular results that have happened in the last few years come from like replay ratios of eight, you know, training eight times as many times as you're actually making an action and these take much longer times or building diffusion models or other things that take days to run to generate hours of gameplay. These are interesting from a technical level, but they're quite far from this solve a problem of acting in the real world.

And the latency in particular is a killer for many of these algorithms because many of these algorithms condition on the action that's taken. They say, I'm building a world model or I'm building a self-predictive representation and it's conditioned on this action that I'm telling the environment. And that works great. It delivers these fantastic abilities on a simulator where it's waiting for you. You get your observation, you make your action, it processes it, gives you your reward and next observation. But if that action is going to take a dozen frames or more to actually have an effect, this simple conditioning on next reward doesn't work at all. It's just going to be random noise. And these spectacular algorithms then pretty much fall apart.

But then there's the other questions like how else might sensing from reality differ and we did this kind of as a stunt but it's been a pretty valuable learning experience for us in a few different ways but this idea of like what would it take to actually play a game like a human plays a game not in simulation and one of the things that I've been telling people that are kind of like oh the you know the singularity is near in a couple years robots are going to be doing everything it's like take your next dancing robot demonst ration and ask it to pick up a joystick and learn how to play a video game that it's never seen before. These are capabilities that are nowhere close to happening in these types of systems. You know, not only does it not have necessarily the dexterity and latency on the loop to kind of do that effectively, but the fundamental problem of learning some of these things is still hard. and

then learning it in an environment with all of the other things going on around it instead of just being fed exactly what you need for this. These are challenges. So, um, this, you know, this pitch and we've got it in the the demo like over there by the curtains and everyone should go look at it. This is a setup where we have a camera, a little robotic servo, joystick stuffed in there and it's running, you know, not particularly exciting RL agent, but it started learning this morning. uh you know and it's up to a kind of mediocreish performance now. If we let it run for a couple days, it'll get to better and better performance. But this was just a test of what do you need to do to make RL agents that work fine in simulation behave okay in the real world. And we will be open sourcing this work, both the uh the physical aspects of it, the 3D printer files for the the joystick controllers and our frameworks

and the the RL agent and all that stuff in there. Uh, we learned enough just from setting this up in public here yesterday that we're going to do a couple more iterations before we wind up making it public, but it'll all get out there eventually. So, some interesting design system design choices on this. Most Atari games are set up so that you have to reach up to the console and pull a little reset lever to restart it, but there's at least a number of them that you can restart just by pressing fire on the joystick after you've died. So, we picked I there was a dozen or so games that we found that could work like that where you could play it just with the joystick and restarting there. So, we didn't have to add another servo up on the the console itself. So, we've got Centipede, Up and Down, Cubert, Defender, Miss Pac-Man, Crawl, Atlantis, and Battle Zone. Now, we had a couple other ones that also worked here that we

had to disqualify for some other interesting reasons related to the score availability. Uh the compute we're running on right now is just a gamer laptop. Uh it's laptops are a little bit frustrating. We were pulling our hair out a little bit last night. Why performance was down because it wasn't fully charged and it was in a power saving mode throttling the GPU, all the miserable things dealing with uh with that type of stuff. But you can get a 16 GB 4090 equivalent GPU in a laptop and that is enough to run sort of most classic deep RL algorithms at a frame skip 4 level. Uh the compute is you know there's lots of fun optimization stuff to do and I'm I always pull it I feel the draw of rabbit holeing down an optimization both in compute and latency end to end. Uh but just a couple quick things. It's like I am a big fan of CUDA graphs where the entire thing that's running there is one CUDA call to the graph and it does

everything. It does the the policy evaluation. It early outs to let the rest of the the real world go on while it's doing training for the next set of things and that's been a pretty efficient way to do things. Uh we've got fairly standard uh kind of live streaming camera setup, but there are some things to watch out for in cameras where uh some of them don't have full 60 frames per second or they do JPEG compression where you really want a low latency uncompressed stream. It does 1080 natively, but you don't need that for Atari. Atari screens are only 160 by 210. They're very low resolution, but there may be some reasons that you want to run higher resolutions for things like identifying tags around the screen, scorebox, and various things like that.

60 frames per second was fairly important. Uh I had hoped initially that we could just use the laptop webcam, just open up the laptop, point it at the screen, and have it look uh have it work there. But that was only 30 frames per second. Not acceptable there. Now, we expected some set of problems with scan out. Anybody that works with cameras and displays knows you've always got the gen lock problem about uh you know your screen is refreshing maybe theoretically at 60 while your camera's at 60 but they're not in phase and they're always slightly drifting. So it's probably more of a problem for uh kind of scrolling games that have full field motion. But the only thing that caused us trouble for in Atari was the big thing that I'll get to again which is this scorebox identification. Sometimes we're looking at a score to tell what the reward's going to be and it'll be split in half and it'll mess up the digit kind of

recognizer for it. So the rectification of looking at the screen, this was an interesting thing. Uh we started off not doing anything. You just point a stick a webcam down, look at a screen, it's got all sorts of other stuff in the background, nothing uh kind of uh routed out and it still learns fine. basic connetss are really surprisingly uh resilient to learning whatever random distortions you might have there. But I if you hope to be able to take this and then reuse it in a different place, uh you need to have be able to just like pull out exactly what you want from the screen. So we went through uh a number of different experiments. You know, we put in uh they're called April tags, these little fidials that robotics people use a lot uh to identify things. and we just kind of stickered those onto a screen around the uh around the game field that we cared about. We could identify those and

then we can pull out a nice rectified image of the screen. And that worked really nicely. You could move the camera around. It kept track of the screen everywhere. And that's sort of what you would imagine your humanoid robot leaning around playing a game would wind up using. But we had miserable problems with a sticker is a reflective thing. the screen is an emissive thing and lighting conditions changed made that fairly unstable and kind of difficult to use. It also kind of wanted higher resolution for that. Uh but at 1080 it worked okay. Um the what we what we want to do and what we're probably going to go back and do after we get back from the conference here is do some more generalized screen detection where it knows what the background field of this game looks like. It should be able to just identify that, rectify everything from that, just point it at a screen. It figures it's a figures everything out.

But that's going to be a little bit more work. On the output side, the first thing to try is just wiring up a little digital IO. All of these RL agents, they just spit out here's your action from, you know, 18 different possibilities, nine different joysticks, button up or down. And the Atari joystick is a nice simple thing. It's not like a modern controller that's sort of a USB spec. It's just pull these lines down to ground and that's what your uh what your joystick does. So, that was pretty straightforward, but again, it's not really real. It's kind of talking to the computer on a computer level rather than a physical level. So, what we've got now is what we call the rootroller, which is a little robotic thing that you literally stick an Atari joystick in.

And it's got three servos for two axes and a fire button. And this does add some additional latency uh which is expected. But the the somewhat surprising thing that we didn't think about is there is a problem with physical things when you move from say like one diagonal axes to another diagonal. There will be intermediate steps along it. Uh that will there will be phantom actions and then the fire button is independent of the the other two axes. So, we've got problems where a game like Atlantis where you're kind of shooting from these turrets on the side of the screen. You push the joystick over and press the fire button. The agents just say, "I want to be firing from this side." And they kind of shoot the shot up that way, but then you run it on the physical system and the joystick is trying to go over, but it hit fire before the joystick got all the way over and it just does terrible. Now,

this is one of those cases where the standard the way people do Atari is uh you just feed in an observation, but this is an example of where you need to be feeding in the reward as well. I mean, not the reward, but the uh the action. It needs to know what you tried to do in the past. So, here it needs to learn don't press fire until a couple frames have gone by after you asked it to go ahead and push over to the right side. And another example I I mention a lot on that is like the old Konami code for old-timers when you remember you're going up up down down left right uh buttons to get cheat code access to some old Nintendo games. That's the type of thing that the standard Atari interface that people use is never going to be able to to actually issue because it doesn't know exactly which buttons it's pressed. It's part of the unobservable state because it makes no impact on the observations.

So, robustness issues. We have worn out servos and a joystick so far with this when you just let these things sit and churn overnight pushing the joystick all over, especially in the kind of way that RL agents early in training tend to play. Lots of kind of vibrating back and forth. So, figuring out did your uh you know, did your RL algorithm just fail because the score is terrible or did you actually wear out one of the buttons on your joystick was another point. So, uh, Kurum did some, uh, really nice timings here looking at the actual latencies, and these seem shockingly high for people that, uh, you know, that are used to thinking in smaller numbers, but with the robot controller, we're looking at 180ish milliseconds end toend latency of having it move from one side to the other. And there can be even more latency in the systems with the camera, and the actual processing. We know that

the the processing is going on at this 15 hertz rate where it's definitely not taking more than 60 milliseconds to process. But this whole chain of things that go on after that and before that add up. And it looks like quite a lot. Now, of course, this is what I did in virtual reality was fighting the good fight and bringing all that down so that the motion to photons latency was only 20 milliseconds, which was kind of the the limit for not causing people to be sick moving their head around and things. And we could have fought that here and tried to get really far down.

Like an interesting thing is we're using this modern system called an Atari 2600 plus and inside it's just got like a little Raspberry Pi running Stella. It is not like a rebuilding of the original Atari system. So just that system adds 50 milliseconds over versus if I dusted off my actual Atari 2600 and we plugged it into a CRT monitor on channel 3. But the point is these are numbers that are roughly humanlike. If you've never done this, you should do search on like human latency testing or click latency testing. Go to a little website and they'll just like flash a color and click as fast as you can. And while there actually are latencies in their test bed that are worth noting, you get numbers like 150 to 200 milliseconds. So these are not crazy numbers uh to be learning with. Now, surprisingly, the thing that turned out to be by far the most problematic aspect of this project was reinforcement learning algorithms

need a reward. They need a reward to chase. So, normally these come from uh like the AL looking at internal memory locations and saying your score went up. I'm going to give you a reward here. Uh but in our situation, we had to look at the screen and figure out what the score was to give a reward for that. And that seems simple, like immnest, haven't we been doing this forever? Isn't this what AI just does? And uh you can take a screenshot. You can ask chatgpt what's the score on this screen? And it'll just tell you. But it turns out doing this with relatively uh little data to sample from in terms of different locations uh and across different games is actually kind of tough. And while this isn't hardcore ML engineering, it's systems engineering that we really aren't quite through. We're still kind of messing around with issues here. Uh we identify a little box of score and run models to

find out the digits. But this morning, we were finding that the tablecloth on the demo table over there was shifting slightly and it was like moving right out of our little score box. Didn't affect the the convolutional net running the model, but our score detector was having problems with this. So, uh, it was it's frustrating that that's still not super great. So, one of the things that we've done that's a little bit halfway is made a custom little dev box. So, it's not an actual Atari, but it's got an Atari joystick port and it can go ahead and throw up these extra fidials on the screen so that you can still have a camera. You still have all the the challenges of dealing with real world lighting and everything, but you it makes the score problem a little bit easier. So, it's it's not as cool as actually running on real hardware, but it lets us kind of grapple with some of those issues. So, uh what we've learned

from this aspect, and this was never like a major goal. few months ago we just kind of decided it's like oh can we put something kind of fun together before upper bound and it's been a worthwhile effort and bringing everybody together feeling like a product push but I you know the connets basically work fine for real world uh issues as far as looking at different amounts of glare different screens learning in place works great we have not necessarily demonstrated transferring to different locations and I expect that there's still a set of interesting little challenges for us to do to say what is the uh invariant representation internally that's not going to care what your uh you know what your lighting conditions and what your monitor uh kind of curvature or anything like that is.

Um the standard TD learning algorithms work fine with extra latency. Uh it varies by game where some games like Breakout with fast balls zipping down at you, they're very sensitive to it and scores drop reasonably precipitously with adding stuff, but uh many other games, Miss Pac-Man's not very latency sensitive. You know, the way a human plays, you push the joystick over well before you're making a decision. But when we've taken uh BBF, the bigger, better, faster Atari 100k kind of uh top dog, it generates really really impressive learning scores in 100k frames. But if we just add a few frames of latency in there, it falls apart. So like our our agent, which again is nothing particularly special here, it's uh you know, it learns much faster than Rainbow, but not as fast as BBF. But add a few frames of latency to BBF and then it's well below that. And I don't want anybody to I want every researcher to

add a delay Q into your uh right before your environment. Just put a little flexible delay Q where you can tune in up and down how many frames you delay. Uh but please nobody solve this by just adding a matching delay Q somewhere else before you get to your action conditioning. you know, we want a better solution like feeding all of your uh your actions in as part of the observation and let it figure out as uh you know, as different things change. So, um the action path problem of like phantom actions in the middle or not having the different uh the different reward axi or the different action axes synchronized, that's a little bit interesting issue. And then uh you know it's still it does decent learning. So the demo over there uh one of them was restarted uh this morning. One of them was running overnight but we had our little issue with the tablecloth moving the score there. But this is small

integer number of hours and it's playing reasonably well. Uh you know it's not human level scores there but if you let them play for days then they do get up there. So the basic idea of the classic RL algorithms do work uh in a robotics environment for for this stuff but and it does look like human level performance is probably feasible but now uh the real thing that we're working on are all of these major RL tasks the things that several of these other to talk talks have talked about the same basic things because these are fundamental problems that are unsolved we're taking our good stab at it and it's going to be in the context of suites of Atari games. So, the uh sequential multitask learning uh when people first heard about Atari games being played, you know, you hear this model plays 57 Atari games and it plays them all well. Some people didn't initially realize that no, each one is

trained. You can make a model that plays Breakout, you can make a model that plays Pong. Even though they're very similar games, they don't play the other games well at all. And interestingly, if you just try to learn in sequence, you learn Breakout for five hours and then you try to you learn Pong for five hours, you go back to Breakout and it's largely destroyed. Uh it picks back up a little faster. Not everything's gone, but it is very much not a human-like system. And people have solved this where if you train them all at the same time altogether, that works out okay.

You can make that work as long as you normalize the scores in a certain way. But again, not how humans work. There's real problems that we have that we don't know how to solve. It's also, in my view, cheating to have a task ID passed in to let the model know. It's like, hey, this is a new task. You should swap out weights, you know, change out layers of your model or whatever. It needs to be a model that just takes what it's given because there's no moral difference between going to a brand new level on a game and going to a brand new game. There's still challenges in the continuous learning that you need to work with. um it is one of the things that I think even many researchers don't fully appreciate is the damage that happens in every update uh to a model that makes continuous learning like this so challenging where the idea that you might train a game for hundreds of millions or billions of frames but every

optimizer step you do changes essentially every weight you know if you've got a feed forward ReLU network they're ch one quarter of the weights are changing but with convolutions and batches every weight is changing which means every response to every observation is changing and you hope that you've done something based on your little mini batch you're improving performance there but you're affecting everything else. Now ideally there would be some positive transfer learning where this is actually helping everything else or maybe at worst it's a random walk but even a random walk takes you into meaninglessness after hundreds of millions of frames. And this is one of the you know this one of the points that like I I argue with rich a little bit about the the virtues of uh replay buffers. You know it's like does a giant replay buffer solve it for us where we have tons of memory. We got 28 terabytes

of NVMe data and storage that we could put things on. Does that just solve it? You know, we're going to run that experiment here within the next few weeks. I see if that works. But there are smarter things that we have to be doing about that. Um, and another challenge of that is if you've got things that you you learned a year ago in sort of subjective agent time, if you're still churning on that and you're learning from that, you're in sort of an offline RL problem. And offline RL is much harder because you're not testing what you're changing. And you can have this kind of coherent fantasy where, wow, I've bootstrapped this into a spectacular thing. Maybe next month when I actually come back and play this, it's going to be good, but more likely it's going to be terrible. So there's some other challenges with that.

And then uh related to that is the transfer learning where all of these RL agents if you just start them up they look like idiots you know like what are you doing you know you can walk over there and look at it playing Miss Pac-Man and even after several hours of training you're like why did you just run into the ghost you know why did you just move back and forth there for no good reason and the model is that these are like an infant opening its eyes for the first time and that's actually then kind of impressive that in a small number of hours it's like the fawn getting up and starting to run with no you know no extra training behind it.

But if you have played 10 games for a month each and you get to a new game, you should be able to play it decently. You should be able to like understand how games work and what you need to do to make progress, not randomly bumble around with the joystick. Uh but this has been almost completely failed for everything. Uh just as I was kind of starting with this or I found OpenAI had done this challenge with uh got to learn fast which was a challenge using Sonic the Hedgehog game kind of a somewhat more modern game than Atari and the idea was they let you train on all the existing games and then they were going to test you on uh a a brand new never-beforeseen map. It's like great, this seems like a really excellent test.

And nobody demonstrated any transfer learning. The winners just ran rainbow and had reasonable tuning just to learn in those 1 million steps. And then uh it seemed exciting where the the gateau agent, the the transformer-based learn to play all the Atari games. First of all, it didn't actually learn to play them, but it learned to mimic pre-trained uh outputs of another agent. But they demonstrated negative transfer learning where if you took a game a model trained on a dozen agents and then you said okay I want to learn a new game through fine-tuning it was harder to do that than to train completely from scratch. So clearly we've got something that we just do not understand. You know this is a a problem. And this is a little bit distinct from the continuous forgetting because even if you were forgetting the old models that you learned you should still learn the new ones faster. They're kind of tied

together, but they're they're a little bit separate problems. So, I'm hoping to get some traction behind getting the research community to agree on a new benchmark based on Atari games that sequences through them sequentially and then tests on like the last cycle of them. And a problem with a lot of benchmarking in in ML, it's it's driven almost all of the advances really chasing number go up. But in most cases, these have been kind of ad hoc specified benchmarks like all the reports for Atari numbers. It's like well somebody makes their own training framework and they work on Atari for a while. They post numbers and there's all of these important little details that I that are often elided even from the research papers or are not reproducible. I think there's a value to making a harness that calls your agent through this extremely simple interface that is just like okay agent you have been told here is your

observation here is your reward give me your action and the environment goes on with no control that you've got over it I think that is the right way to kind of address these problems going forward and then all the details about it should have sticky action so it's not a deterministic thing it should use the full action set so you can go between all the different different possibilities ities and we're going to do a bunch of these experiments uh kind of in the you know the coming months but I'm hoping that we can actually get some community buy in and draft some more people to be kind of benchecking us against other things.

Sparse rewards is one of the other major things where in real life almost never do we have the density of rewards of a video game where it's saying score go up score go up sometimes multiple times a second and classically the Atari games that everybody had the most problems with they called them the hard exploration games like Pitfall and Monzuma's Revenge and those were games where you might play for minutes without getting any reward signal and if all your agent's doing is bumbling around until it gets it that's how you got some of those runs that required 20 billion frames to start delivering really good scores. So, there's something important that needs to be done there. And it's noteworthy that even on a video game, humans usually aren't looking at the score nearly as much as an RL agent does. If you actually play games when you die, you look at your score. You know, when it's game over, you look at

your score to see how you did. Maybe if something unusual happened, but usually you're concentrating on the action. The score is something that's going up there kind of off in the corner. And historically, it's interesting where uh my old video game Doom was the first major game that actually action game that didn't have a score. Up through there, all of our previous games, the Commander Kees and Wolfenstein, we were arcade people. So, we had lives and scores. You know, you ran out of lives. It's like you had to put another quarter in. And it was kind of funny that it took that long to realize that these games at home I you know you own the game and maybe that's not the best way to direct play but modern games generally don't have scores like that.

So we need to get more towards these intrinsic rewards and curiosity and letting the game or letting the agent figure out exactly what it's trying to pursue. So uh exploration is another one where the classic DQN approach is epsilon greedy like every hundred actions pick a random action. You know this is clearly not the way any human works. Now policy based action uh policy based models work a little bit better where you can avoid doing some of the tragic things like falling off the side on cubert where sometimes actions are super super important. every time you're sitting on the edge of a cubert level, you hit the wrong action, you die. As opposed to Pong, where you're moving up and down in a kind of continuous way and it doesn't matter that much. But, uh, modern game controllers, like picture of an Xbox controller there, Atari gets by okay with a discrete action space of 18. It's got nine directions for the joystick,

one button. Every button you add doubles the number of actions there. Every new joystick you add multiplies it by nine. So even disregarding the fact that some of these are analog inputs. A modern Xbox controller has over a million actions possible. Clearly that's not going to be a discrete action space and you're not going to be modeling it that way. So we've done experiments with factorizing the actions into like each axis which kind of addresses the issue with uh the robo troller of different different latencies for that but it's an interesting thing. But then time scales are another interesting thing where traditionally Atari games work in this frame skip four regime where you make one action you repeat it four times and this causes the characteristic buzzing of the reinforcement act learning agents where they're just kind of moving back and forth. Humans obviously have more directed goals like I want to get to

that edge of the level. I want to go grab that item. And uh this seems to be where options in RL research apply, but I've yet to see uh a useful application of that to Atari games. You know, we're going to we're going to try. Uh one thing that Atari is unfortunately not good at is we need to have recurrence in our models. You know, the brain is a recurrent neural network. We digest prior information into a a more limited state that we kind of maintain and do continual updates on. Uh, but for Atari, since they're mostly play fields, the standard practice of taking a handful of frames and just providing that history works well enough. We're still not seeing any real benefit from introducing recurrence. We're probably going to have to find some some different tasks for that.

Uh, I'm almost I'm pretty much out of time here. I've got all these things that are actually nitty-gritty technical details that uh I will probably wind up posting these and maybe talking over it uh sometime later, but uh we can actually go to questions and let that direct now as the timer ticks down to zero. So, So, do we have a microphone out?

Hi, John. Thanks for the talk. Um, I'm wondering about the benchmark that you proposed because one thing that I have been very critic about the Atari 100K is the fact that people start to optimize for the length of the training process, right? like I can crank up my step size and if I crank up the step size I know that I will hit a good thing by 100k but it doesn't mean that if I train for 200k this is going to work and I really like the vision that you laid out here but it seems that this benchmark when you start thinking about doing the sweeps and then you have this very short time period focusing on learning fast aren't you opening up yourself to this type of overoptimization that you've been so careful to lay out the places where people to cheat. And this has been always my complaint about the Atar 100K.

It feels that a little bit too easy to cheat. And it also ties a little bit to exploration, right? Like because then in the short time frame, it ends up being much more a matter of luck on whether you hit the the the bullseye within that time frame or not. So you're also opening up yourself to the variance of this performance. Yeah. So I would love to hear your thoughts about this type of stuff. Yeah. So those are very real issues and I and these are these are things that I want to have discussion with everybody about because normally the variance is huge on there and we would like to say like train for longer times but it still needs to be something that the experiment velocity needs to be fast enough where you need to actually be able to do experiments. You can't have something where it takes a month to tell whether your your idea is good. So, this idea of maybe eight games, three cycles.

It's roughly the compute of running one Atari 100K. I'm open to people saying, "No, we can handle more than that." But I suspect a bunch of people working on things are like, "That's already a little bit of a stretch to tie up resources because, as you know, you need to do like 20 different seeds just to have some statistical certainty about it. Your single run can have wild variance in the different numbers." And as far as the actual things like I, you know, like I don't love BBF's model resets every 20k uh because your performance plummets at that point there. One thing that I want to have is no explicit evaluation phase. I like I do think in the in the big benchmarking I think Mui did it right where there was there were full action set sticky actions and it was not a dedicated evaluation. It was like the last 200 episodes in the training. So thinking something like that where it's the last

set of episodes. So if you're doing model resets, you're going to get some zero scores kind of thrown into your average to at least partially penalize that. Um, and as far as the exact switching, no, there's like a question of do you switch exactly at episode boundaries or do you truncate? Because an interesting an interesting thing happens. If you just truncate randomly and you don't specifically handle that, the agent will it will seem to the agent like a bolt of thunder from the blue just destroyed them. And you we saw some significant negative training uh when we were doing that, but you can pass in an explicit termination signal. So I don't know the question of I would think letting it train complete episodes so they're not aligned with say ch changing every 100k steps is probably better. But I am open to this. But I do think even a bad benchmark is better than no benchmark.

You know, you got to measure something. If it's just random conclusions and and abstracts on things, if it's not measured against something else, I think a lot it's too easy to have miragages of uh of value. So, I'd rather have people gaming the benchmark than not running the benchmark at all. So, buyin is more important to me than making it sort of cheat proof. And then you just use community appribation. It's like, well, that's totally a cheat. Nobody should pay attention to it. Uh yeah, thank you for pulling the curtains back a little bit. You you talked about uh the difference between like explicit scores versus intrinsic reward. You have a lot of experience building interactive systems that engage humans intrinsic rewards, things that we would play without even looking at the score. I'm curious if you have thoughts on like differences between designing machine intrinsic rewards versus what

engages humans. Are there things that we shouldn't transport from human curiosity? So that's a very interesting question because I mean humanity is our existence proof of the intelligence that we're trying to mimic. So the question of how many of humanity's quirks do we emulate to try to show that and I think there's some obvious ones that you do mimic like this idea of you you reward agency on some level. I mean knowing that you're controlling something and you're making something happen. And there's something about visual field effects like the most, you know, the more things that happen, the little pixel twiddling is less rewarding than the gigantic explosion over the whole screen. But when you get down to some of the specifics, it's less clear. Like there are sort of, you know, people reward hack the human being, the human mind, and things like online gambling and the different things where you can sort of

work out your reinforcement schedules to maximize that. You know, that's probably not ideal. I think it's probably best to let the final score be the main driver where say it's a good intrinsic reward if it improves the final score. And if we're in this, you know, if we make it all sparse rewards, if you don't give any any rewards there, it might learn how to read the score and decide score going up is a good thing. And I'd be perfectly happy with that if the model found that out or if it hadn't found some other way. But score in the end is probably the right metric for us to be looking at. And if it doesn't improve score, maybe it's interesting theoretically, but it's probably not uh an actual good reward for it.

Questions about this episode

What is "John Carmack (the creator of Doom) Reveals AGI Future: Robots, Videogames and AI Agents..." about?

In this episode, John Carmack, creator of Doom and former Oculus CTO, pulls back the curtain on Keen Technologies, his secretive AGI research venture. He traces his path from pioneering first-person shooters and GPU adoption to VR systems engineering, and explains why he left Meta to pursue artificial general intelligence. Carmack makes the case that large language models, despite their…

What is mentioned in this Wes and Dylan episode?

39 people, works and ideas are mentioned, including OpenAI, Rainbow, Keen Technologies, ID Software, Nvidia, Armadillo Aerospace. Each one on this page links to its own page with every episode that mentions it.

How long is "John Carmack (the creator of Doom) Reveals AGI Future: Robots, Videogames and AI Agents..."?

The episode runs 51 min; the transcript is about 9,523 words in 62 timestamped sections.

More from Wes and Dylan

About this transcript: the episode's spoken words as captioned on YouTube, split into timestamped sections. Reclip.Me locates every person, work and idea in the text, keeps the time it is first named, and quotes the words around it verbatim. Last updated July 9, 2025.