Transcript

Google’s Genie Turns Images Into Video Games | “Unlimited Interactive Worlds”

From Matthew Berman · Episode page · Watch on YouTube

· 22 min · 3,107 words · 19 timestamped sections

This is the full, timestamped transcript of "Google’s Genie Turns Images Into Video Games | “Unlimited Interactive Worlds”", an episode of Matthew Berman published February 28, 2024 (22 min, about 3,107 words). It mentions 19 people, works and ideas, including Dr. Jim Fan, Simulation Theory, Gemini 1.5 Pro, Google LLC, Nvidia.

What is discussed, and when

  1. – 5:00: AAA video game production company, Artificial Intelligence, Google LLC
  2. – 10:00: Dr. Jim Fan, Simulation Theory
  3. – 15:00: Dr. Jim Fan, The Hollywood Reporter, 11 Labs, Nvidia
  4. – 20:00: Gemini 1.5 Pro, 3.js, Apple Vision Pro, Edgar Hond
  5. – 21:44: Google LLC, Nvidia, Simulation Theory

Notable quotes from "Google’s Genie Turns Images Into Video Games | “Unlimited Interactive Worlds”"

  • Imagine this you're a AAA video game production company you're responsible for some of the most beloved games in the last 30 years but your industry has been struggling players expect more.
    about AAA video game production company · at
  • As yesterday that really show a glimpse into what the future of content will be introducing Google's Genie generative interactive environments it's a foundation World model…
    about Google LLC · at
  • Generalist agents now we've been talking a lot about embodied agents recently especially with the work from Dr Jim fan out of Invidia a lot of my predictions from the beginning…
    about Dr. Jim Fan · at
  • Model World model and what it's all headed to and what I'm going to talk about later in this video is simulation Theory and it's all going to tie back to that something that I…
    about Simulation Theory · at
  • Rules of that world then we have the work of Dr Jim fan and his team at Nvidia who are now generating massive amounts of synthetic data using their Isaac Jim product and other…
    about Nvidia · at
  • About that through this video and future videos but so far we've only talked about things that are visual 11 Labs has taken the Sora videos and run it through their own models…
    about 11 Labs · at
  • Industries are already realizing what's around the corner here's an article from The Hollywood Reporter Tyler Perry puts $800 million Studio expansion on hold after seeing open…
    about The Hollywood Reporter · at
  • Talk about in this video is something that I've been playing around with a lot this week I got access to Gemini 1.5 Pro with a 1 million token context limit now a million…
    about Gemini 1.5 Pro · at

Who and what is mentioned in this episode?

People

  • Dr. Jim Fan — Person · 3 mentions · first at
  • Tyler Perry — Person · 1 mention · first at
    Industries are already realizing what's around the corner here's an article from The Hollywood Reporter Tyler Perry puts $800 million Studio expansion on hold after seeing open…
  • Edgar Hond — Person · 1 mention · first at
    Formed opinions and they displayed emergent human behavior incredible so the author of AI reality TV Edgar Hond and I apologize if I'm pronouncing your name wrong took that as…

Books & papers

Companies & orgs

Products & tech

  • Gemini 1.5 Pro — Product · 3 mentions · first at
  • 3.js — Product · 1 mention · first at
    Is within the 1 million tokens of context that Gemini 1.5 Pro allows they loaded up 3. JS a 3D Library built in JavaScript the entire library and not just the library all the…
  • Apple Vision Pro — Product · 1 mention · first at
    Back it by Insane computing power which we probably don't have yet and then put on your Apple Vision Pro or VR goggles of choice and you are in a world that you are painting…
  • Genie — Product · 1 mention
  • Isaac Gym — Product · 1 mention
  • Sora — Product · 1 mention

Concepts & topics

  • Simulation Theory — Concept · 3 mentions · first at
  • Artificial Intelligence — Concept · 1 mention · first at
    And then all of a sudden Genie makes it playable and this is another piece of artificial intelligence this is not hardcoded this is not somebody taking that image splicing it…

Full transcript with timestamps

imagine this you're a AAA video game production company you're responsible for some of the most beloved games in the last 30 years but your industry has been struggling players expect more games cost more to make and your margins are shrinking and one day you wake up and realize AI is about to completely disrupt your industry all of a sudden it's possible for every video game player to Simply describe a video game that they want to play and AI makes it in seconds does that seem Fantastical something that might happen in the next 30 or 50 years think again I'm going to show you that we're actually incredibly close to that future and video game companies aren't the only ones scrambling to adapt right now movie studios are as well content for an audience of one is right around on the corner let me show you how and stick around until the end because what happens after AI takes over all media is

truly the work of Science Fiction if you've been on YouTube over the last 2 weeks you must have seen all of the incredible videos coming from open ai's new textto video product Sora Sora allows you to Simply describe with natural language a scene that you want to see and it outputs up to a minute long video of that scene what made Sora unique is the fact that it had a very solid understanding of physics meaning it could track objects through the entirety of the video keeping them consistent and having them interact with the world as if it were reality the videos displayed are hyper realistic the movements the way hair Falls the way physics interacts all looked incredibly real it blew a lot of people's minds and immediately video game developers and movie studios were worried and rightfully so and this only happened two weeks ago so I'm going to show you a few different technologies that have come

out around the same time and as recent as yesterday that really show a glimpse into what the future of content will be introducing Google's Genie generative interactive environments it's a foundation World model trained from internet videos that can generate an endless variety of playable that's the key word playable worlds from synthetic images photographs and even sketches you simply give it a picture something that can be playable and then all of a sudden Genie makes it playable and this is another piece of artificial intelligence this is not hardcoded this is not somebody taking that image splicing it up understanding who the character is what the platforms are how to jump on them this is purely here's an image output something playable and that's what we're seeing here it is incredible and as I continue to show you all of these Innovations you are going to clearly see what the future

of video games and all media is going to look like so let's take a closer look at Genie Genie can be prompted with images it has never seen before such as real world photographs or sketches enabling people to interact with their imagined virtual world essentially acting as a foundation World model now picture this you prompt Sora for some Fantastical or even hyper realistic world you pass it through Genie and and then all of a sudden you have a video game the foundation for that type of technology is here today being able to have a fully production ready version of this technology seems like it is right around the corner this is possible despite training without any action labels instead Genie is trained from a large data set of publicly available internet videos we focus on videos of 2D platformer games and Robotics but our method is general and should work for any type of domain and is scalable to

ever larger internet data sets so in these examples nothing is labeled Genie figures out what parts of the image are let's say the platform and the character can't go through the platform so it is actually figuring out the physics of the game with no information other than the image itself here are some more examples these are the the input images that we're seeing everything from kind of slightly 3D Platformers to 2D Platformers to completely 3D worlds and then all of a sudden what we're seeing below it is the live video game but not only that it can take rough sketches children's drawings and make them into playable games and real world images so these are just pictures of things that happened in the real world and then all of a sudden we have a playable video game from them so all of a sudden anything can become a video game and the most interesting part it translates to robotics really well so let's see what

it says here finally while we have focused on results from Platformers on this website Genie is a general method and can be applied to a multitude of domains without requiring any additional domain knowledge we have trained a smaller 2.5 billion parameter model on action free free videos from rt1 and if you're not familiar rt1 is a paper that described how robots can and will be controlled in the future by Ai and specifically large language models as was the case for Platformers trajectories with the same latent action sequence typically display similar behaviors this indicates Genie is able to learn a consistent action space which may be amenable to training embodied generalist agents now we've been talking a lot about embodied agents recently especially with the work from Dr Jim fan out of Invidia a lot of my predictions from the beginning of the year especially around robotics seem to be

coming true at a rapid Pace we are seeing robots advance so quickly in their capabilities because of synthetic data and because of teleoperation and last before we dive into the paper Genie can also simulate deformable objects like a shirt a challenging task for human designed simulators that can instead be learned from data now I want to quickly go through the genie paper because I find it to be incredibly fascinating so here is how it works whether you have a text to image a handdrawn sketch or a real world photo you pass it through Genie and then all of a sudden you have a playable Game Genie is capable of converting a variety of different prompts into interactive playable environments that can be played easily stepped into and explored it is trained in an unsupervised manner from unlabeled internet videos and it can be prompted to generate an endless variety of action controllable Virtual Worlds

described through text synthetic images photographs and even sketches now the most surprising part to me this model is only 11 billion parameters and and can be considered a foundation World model we keep hearing these words Foundation model World model and what it's all headed to and what I'm going to talk about later in this video is simulation Theory and it's all going to tie back to that something that I am fascinated with here it says early signs indicate video generation will be yet another Frontier with recent results suggesting that such models may also benefit from scale and that is what we're seeing with Sora Sora is not using truly new technology it is using the same Transformer underlying technology that large language models are built on so it seems like we still have plenty of Headroom to grow into the Transformers technology and I love this sentence what if given a large Corpus of

videos from the internet we could not only train models capable of generating novel images or videos but entire interactive experiences and they didn't say video games they are saying interactive experiences we propose generative interactive environments a new paradigm for generative AI whereby interactive environments can be generated from a single text or image prompt 200,000 hours of publicly available internet gaming videos and despite training without action or text annotations is controllable on a frame by frame basis via a learned latent space action at 11 billion parameters Genie exhibits properties typically seen from Foundation models so impressive so now let's review what we have so far we have Sora capable of generating hyper realistic worlds from text prompts then we have Genie being able to take images currently images but probably soon video and convert them into actual interactive

worlds with a real understanding of the rules of that world then we have the work of Dr Jim fan and his team at Nvidia who are now generating massive amounts of synthetic data using their Isaac Jim product and other releases that they've published recently and they are creating AI agents that can interact in these worlds so you have a world that you can create with text you have the Genie model that can convert the world into being fully interactive you have the work Dr Jim fan is doing and putting embodied agents in these worlds all of a sudden if you're able to simulate reality what is the difference between that simulated reality and the true reality I want you to keep thinking about that through this video and future videos but so far we've only talked about things that are visual 11 Labs has taken the Sora videos and run it through their own models to create realistic audio for each of the worlds created

with Sora let me show you a few examples now in a place beyond imagination where the Horizon kisses the heavens one man dares to Journey where few have ventured armed with nothing but his wit and an unyielding Spirit he seeks the answers to Mysteries that lie beyond the stars so now again let's review what we have we have full immersive video created by just a prompt we have full sound created by just a prompt a deep understanding of the physics of these worlds then we can convert them to being full interactive worlds then we can place AI agents with a full understanding of the world in these worlds this is mindblowing now bringing it all the way back to video game makers they must be nervous because we are not far away from video games movies television shows for an audience of one I simply describe what I want to see what kind of game I want to play and within seconds that is created for me no more need for

teams of hundreds or or even thousands to create AAA games incredible production budgets are no longer necessary what are these industries going to do and the top people in these industries are already realizing what's around the corner here's an article from The Hollywood Reporter Tyler Perry puts $800 million Studio expansion on hold after seeing open AI Sora quote jobs are going to be lost so Tyler Perry literally stopped the expansion of his massive Atlanta based Studio because he saw the Sora samples and that was enough for him to know that's the future and now he's probably thinking how do I make this technology myself or how do I license it from open Ai and I know people probably were talking a lot about what would be possible in the future but actually seeing what was possible with the Sora demo and now with the genie demo paints another picture entirely quote being told that it can do all of

these things is one thing but actually seeing the capabilities it was mindblowing and he said that in an interview with The Hollywood Reporter noting that his Productions might not have to travel to locations or build sets with the assistance of the technology he also is asked are you currently implementing AI in any of your Productions and and or do you plan to do so in the near future quote I just used AI in two films that are going to be announced soon that kept me out of makeup for hours in post and on set I was able to use this AI technology to avoid ever having to sit through hours of Aging makeup and that's just one slim use case he is realizing that the entire movie right around the corner is going to be made by AI another thing I want to talk about in this video is is something that I've been playing around with a lot this week I got access to Gemini 1.5 Pro with a 1 million token context limit now

a million tokens of context opens up Incredible use cases but why am I mentioning this within the context of this video well I want to show you an example where a large language model is able to create a virtual environment simply by describing what you want want created so let me show you how cool this is within the 1 million tokens of context that Gemini 1.5 Pro allows they loaded up 3. JS a 3D Library built in JavaScript the entire library and not just the library all the code that runs it but also all of its documentation and specific examples all in a single prompt and then they start asking it questions so in this example they ask what controls the animations on the littlest Tokyo demo here's the littlest Tokyo demo and it says what controls the animations and it's asking specifically what are the pieces of code that can control that demo then he says show me some code to add a slider and so almost

instantly it adds the code for a slider and if you look there it is there's the slider that can control the speed of the trolley and that's only happening because of the massive context window next he passes in an image and says where can I find the code for this demo and there it is in example slash and then it provides the HTML for that example page how can I modify the code to make the terrain flatter there's some more code so picture this you have Sora creating amazing virtual environments you have Genie converting those environments into actual playable games and then you have something like Gemini 1.5 Pro where you can just say natural language and control anything within the environment now put this all together back it by Insane computing power which we probably don't have yet and then put on your Apple Vision Pro or VR goggles of choice and you are in a world that you are painting for yourself and at that point

if you're able to create hyperrealistic worlds that you can control in real time and interact with in real time what's the difference between that and reality now I want to show you one last thing before we touch on simulation Theory this is a project that I recently came across called air.tv and let's just read how it describes it itself welcome to AI reality TV the first TV show where the characters are all AI now this project was clearly inspired by the Stanford paper that came out last year that I loved which was titled generative agents interactive simulakra of human behavior and a quick note about what this paper was about if you didn't watch my video about it basically the researchers created this virtual world very small very basic put 25 AI agents based on chat GPT into this world and allowed them to live their lives those AI characters built relationships they formed habits they

formed opinions and they displayed emergent human behavior incredible so the author of AI reality TV Edgar Hond and I apologize if I'm pronouncing your name wrong took that as inspiration and created AI reality TV so you open up AI reality TV and you watch these characters in real time live their lives as if it were a reality show let's take a look so we can easily scroll around this little town it's pretty small to begin with but it looks really cool and of course the graphics are pixel 2D very basic but I want you to picture taking this concept and expanding it to everything else that we've been talking about in this video having 3D worlds instantly created for you having AI agents that you can place in these worlds and live their lives and then you can join the world and interact with it and anybody in it so each of these characters is powered by AI this is running 24 hours a day you can click

on any character to see what they're up to so this guy is named ai. ronac name is Vision bro he is truly Visionary thinks about tech future of life in 10 1,000 years he likes to read Naval ravikant Steve Jobs and Richard fman here are the last five tasks he's done Tech wave watching Tech Time Machine Tech future Quest so these are the actions he's taking in the game and here's his last conversation so these AI characters from within the game can have conversations and here's one live right now these two characters are having a conversation right now so here it is Tammy and Bella are having a live conversation powered by AI right in front of our eyes truly incredible if you want to check this out I'll drop a link in the description below it seems like a very very cool project okay let's put it all together now I've been talking about simulation theory on and off for the last few weeks and it seems

like we have all of the tools to achieve simulation Theory let's go through through it one more time we have entire Virtual Worlds hyper detailed that can be created with Sora we have ai agents that can be placed into the world powered by gy fan and nvidia's Technology we can then control the worlds ourselves powered by Google's Genie and all of this Can Be Imagined just by describing what we want so at a certain point these Virtual Worlds will become full simulations of any reality and at that point if we can't tell the difference between the simulated environment and our reality what's the difference if you liked this video please consider giving a like And subscribe and I'll see you in the next one

Questions about this episode

What is discussed in "Google’s Genie Turns Images Into Video Games | “Unlimited Interactive Worlds”"?

In order: 0:00 AAA video game production company, Artificial Intelligence, Google LLC; 5:00 Dr. Jim Fan, Simulation Theory; 10:00 Dr. Jim Fan, The Hollywood Reporter, 11 Labs, Nvidia; 15:00 Gemini 1.5 Pro, 3.js, Apple Vision Pro, Edgar Hond; 20:00 Google LLC, Nvidia, Simulation Theory.

What is mentioned in this Matthew Berman episode?

19 people, works and ideas are mentioned, including Dr. Jim Fan, Simulation Theory, Gemini 1.5 Pro, Google LLC, Nvidia, The Hollywood Reporter. Each one on this page links to its own page with every episode that mentions it.

How long is "Google’s Genie Turns Images Into Video Games | “Unlimited Interactive Worlds”"?

The episode runs 22 min; the transcript is about 3,107 words in 19 timestamped sections.

More from Matthew Berman

About this transcript: the episode's spoken words as captioned on YouTube, split into timestamped sections. Reclip.Me locates every person, work and idea in the text, keeps the time it is first named, and quotes the words around it verbatim. Last updated July 2, 2025.