Sparks of AGI, Jagged Frontiers, AI forms "Internal Models" and the Dark Forest Hypothesis...

What is discussed, and when

1. 0:00 – 10:00: Dylan Curious, Sparks of AGI, Watson
2. 10:00 – 20:00: Agent Smith, Attorney General, Digital Twins, Replicator
3. 20:00 – 30:00: World government, Authoritarian regimes, Colossus, Memphis
4. 30:00 – 40:00: Dark forest hypothesis, Known universe, AlphaFold, Neural link
5. 40:00 – 50:00: 150 characters, Attorney General, Bioengineering age, Black Mirror
6. 50:00 – 1:00:00: Democracy, Digital Twins, Actuarial tables, Authoritarian regimes
7. 1:00:00 – 1:05:38: Black Mirror, Ethics, Jeffrey Hinton, Wes Roth

Notable quotes from "Sparks of AGI, Jagged Frontiers, AI forms "Internal Models" and the Dark Forest Hypothesis..."

• Apex intelligence in the planet. We've been the smartest things on the planet for a long time. Hi, I'm Dylan Curious from the YouTube channel Dylan Curious and I am really…

  about Dylan Curious · at 0:59

• Things. And this is where it gets a little fuzzy for me. Like I can imagine a government a president AI that does a better job than any human that we vote for. So we kind of…

  about Attorney General · at 12:52

• I'm not going to talk to my digital twin. I'd like to just do this on my own. I mean that's very thoughtprovoking.

  about Digital Twins · at 15:03

• More cooperation. There was some talks about maybe even some sort of a like a world government organization that's like top down making sure that nobody's developing AI. I mean…

  about World government · at 20:15

• Force cuz the number of variations that it can fold into is like more than the number of atoms in the known universe, right? So it's like this like insane number. So you can't…

  about Known universe · at 32:10

• So it's like why haven't we contacted anything out there? The scariest outcome is kind of like the dark forest hypothesis is that like just everyone out there figured out that…

  about Dark forest hypothesis · at 39:25

• I mean like I said it's sort of vague like sometimes my mind goes into something like a black mirror like well then you just build on a planet some kind of computer and just you live in the computer because it simulates it.

  about Black Mirror · at 40:39

• Yes, I would be extremely careful about that. I guess is the do you see a way for democracy to continue in the age of like hyper intelligent agents? Yeah, I guess maybe it…

  about Democracy · at 51:21

Who and what is mentioned in this episode?

People

• Scott Aaronson — Person · 4 mentions
• Demis Hassabis — Person · 3 mentions
• Ilia Sutskever — Person · 2 mentions
• Sam Altman — Person · 2 mentions
• Wes — Person · 2 mentions
• Dylan Curious — Person · 1 mention · first at 1:02
• Agent Smith — Person · 1 mention · first at 15:51
  Original Matrix movies. Remember when Agent Anderson is that his name or agent Smith? Agent Smith. Yeah. the he goes, at first we created the perfect utopia for you humans and…
• Jensen Huang — Person · 1 mention · first at 47:07
  What's interesting to me is we're entering a new kind of era now where before and this is I Jensen Huang talked about this, Dei Hasabis talked about this the idea that before…
• Dave Shapiro — Person · 1 mention · first at 53:31
  Can? I can see that happening. So the question at that point is, do we allow it to take over? And so Dave Shapiro asked his audience, I actually asked my audience, most people…
• Jeffrey Hinton — Person · 1 mention · first at 1:00:42
  Mean, I don't know. I try to keep an open mind. So, a lot of people like Jeffrey Hinton, lots of others, they're sort of suggesting that these things could be have some sort of…
• Wes Roth — Person · 1 mention · first at 1:05:02
  This for the first time, there's the first half of the podcast up on his main channel. You can go search that at Wes Roth. You can also subscribe because he does amazing video content all the time.
• Aubrey de Grey — Person · 1 mention
• Daario Amodei — Person · 1 mention
• Eliezer Yudkowsky — Person · 1 mention
• Elvis — Person · 1 mention
• Ethan Mollick — Person · 1 mention

Books & papers

• Black Mirror — Book · 2 mentions · first at 40:42
• Sparks of AGI — Book · 1 mention · first at 5:12
  Four months later you had GPT4 and that's when Microsoft published the paper sparks of AGI. So they were saying this is kind of a prototype AGI. Reading that paper that was…
• Star Trek — Book · 1 mention · first at 18:21
  It's like we have to get used to the Star Trek replicator world and not just leaning on it. You can't just get addicted to the holiday, you know? You have to somehow know your own weaknesses and like put some artificial fences.
• The Culture series — Book · 1 mention · first at 36:29
  More from like how it's in science fiction where like both Elon Musk Demi Sasabis they both point to the culture series those series of science fiction books as what they see a…
• Player of Games — Book · 1 mention · first at 36:44
  Hoping for that's what they're seeing great series of books for people that haven't tried it I would do the player of games that's probably one of my favorite great one to get into.
• 1984 — Book · 1 mention
• Social Simulacra paper — Book · 1 mention
• Soma — Book · 1 mention
• The AI Revolution (Parts 1 and 2) — Book · 1 mention

Companies & orgs

• Wait But Why — Company · 1 mention · first at 29:38
  Before even chat GPT and LLMs, there was a blog post on wait but why and it was called there was AI revolution parts one and two. He was explaining what intelligence is. So we kind of think of intelligence as this line that just kind.
• Isomorphic Labs — Company · 1 mention · first at 47:37
  Constructing life Now we're beginning to build stuff. Deep Mind and their spin-off isomorphic labs, they are, they're building new proteins that like bind to other proteins so that we're able to control their functions.
• DeepMind — Company · 1 mention · first at 48:33
  Want to run into some threeeyed whatever on your hike. So, but yeah, for sure. Google DeepMind has made it a mission. For example, they said one of the things that they want to…
• Google Deep Mind — Company · 1 mention · first at 48:33
  Want to run into some threeeyed whatever on your hike. So, but yeah, for sure. Google DeepMind has made it a mission. For example, they said one of the things that they want to…

Products & tech

• Flying cars — Product · 1 mention · first at 41:25
  Like some people have mentioned it's like you know we always thought we all thought that we would have flying cars but instead we have 150 characters or whatever right so the…
• Plastics — Product · 1 mention · first at 48:52
  So, for example, a bacteria that eats plastics, we can put in the ocean to clean up all the pollution. That would be incredible, right? or a bacteria that eats something that we have a lot of and excrete some sort of a bofuel.
• Nano robot — Product · 1 mention · first at 49:15
  There's active efforts in that direction. So that to me is absolutely phenomenal. Like a just little tiny nano robot, not robots, but actual biological life form that fixes…
• Biofuel — Product · 1 mention
• ChatGPT — Product · 1 mention

Concepts & topics

• AGI (Artificial General Intelligence) — Concept · 7 mentions
• ChatGPT — Concept · 6 mentions
• Step function (intelligence staircase) — Concept · 6 mentions
• Artificial superintelligence — Concept · 4 mentions
• GPT-4 — Concept · 4 mentions
• LLM — Concept · 4 mentions
• Digital Twins — Concept · 3 mentions · first at 15:03
• Democracy — Concept · 3 mentions · first at 51:25
• Athel GPT — Concept · 3 mentions
• Pain — Concept · 3 mentions
• Attorney General — Concept · 2 mentions · first at 12:55
• World government — Concept · 2 mentions · first at 20:20
• Authoritarian regimes — Concept · 2 mentions · first at 21:00
  Got to talk about P Doom. We also got to talk about what happens if an authoritarian regime gets access to this technology before everybody else. So he's there are multiple bad…
• Known universe — Concept · 2 mentions · first at 32:15
• Dark forest hypothesis — Concept · 2 mentions · first at 39:30
• Autoencoders — Concept · 2 mentions
• Escape velocity (longevity) — Concept · 2 mentions
• Pre-crime / predictive policing — Concept · 2 mentions
• Ticks — Concept · 2 mentions
• Watson — Concept · 1 mention · first at 6:19
  Programming something and learning something. For me, that first moment was when I was watching Watson. This was a long time ago. And it was learning the different ways you…
• Replicator — Concept · 1 mention · first at 18:21
  It's like we have to get used to the Star Trek replicator world and not just leaning on it. You can't just get addicted to the holiday, you know? You have to somehow know your own weaknesses and like put some artificial fences.
• Colossus — Concept · 1 mention · first at 25:23
  Much looking forward to because they're building that giant data center, the Colossus or whatever it is. Yeah, the Memphis one. I think it is. That's where it is. Yeah, it's…
• AlphaFold — Concept · 1 mention · first at 32:29
  We fed it into AlphaFold and then it started spitting out predictions for other proteins that we haven't sort of discovered or knew the shape of.
• Terraforming — Concept · 1 mention · first at 37:17
  We're terraforming planets. We're building incredible stuff in space.
• Neural link — Concept · 1 mention · first at 38:52
  Becomes super intelligent, then maybe it'll say like what's the point of and maybe we're augmented with a neural link type thing and we're well, what's the point of like this…
• 150 characters — Concept · 1 mention · first at 41:27
  Like you know we always thought we all thought that we would have flying cars but instead we have 150 characters or whatever right so the digital thing evolves so much faster…
• Jagged frontier of intelligence — Concept · 1 mention · first at 42:22
  Made this concept that I think more people should be aware of. He described it as the jagged frontier of intelligence or the jagged frontier of abilities. And so basically the…
• Microbiomes — Concept · 1 mention · first at 45:09
  Strides every day. It seems like it's doing things with genetics and microbiomes and the vyro that would have blown my mind like 5 years ago. And it's leading me to think that…
• Plaque build-up in the brain — Concept · 1 mention · first at 46:11
  A couple different things that tends to kill most of us like cancer heart disease there's some plaque build up in the brain like these things that if we're able to fix those…
• Bioengineering age — Concept · 1 mention · first at 47:27
  Viruses or this and that and we're slowly entering kind of the bioengineering age where we're like constructing life Now we're beginning to build stuff. Deep Mind and their spin-off isomorphic labs, they are, they're building new.
• Gene drive — Concept · 1 mention · first at 49:31
  Maybe g maybe gan drives would be another place we could start like do you familiar with the idea of a gene drive like that it's in the sperm and gamet cells or whatever so it continues through evolution, right?
• Malaria — Concept · 1 mention · first at 49:52
  They keep mating and they make every mate that the baby comes from have no ability to spread malaria and then quickly it's like 2 4 8 16 and you can kind of knock out the whole…
• Rank choice voting — Concept · 1 mention · first at 52:39
  Should be like how do you want the world to look like? Yeah. Like I've thought about rank choice voting before and how sometimes the best candidate is like neither of the two…
• Centralized government — Concept · 1 mention · first at 53:04
  Beginning to talk about this idea that, with AI, you could potentially have kind of a centralized government, even though that kind of like has failed before likely because it…
• Latent space — Concept · 1 mention · first at 57:38
  Similar to how the neural nets they might have certain thoughts or whatever that they have in the latent space. So they're thinking about something that we can't readily…
• Actuarial tables — Concept · 1 mention · first at 59:30
  So similar to you know how insurance works it's like these what is it the actuarial tables or whatever like they're statistically really accurate.
• Ethics — Concept · 1 mention · first at 1:00:07
  That means we're following subscript obviously. So what about ethics and morality? How do you say please and thank you every time you use a large language model? Do you do it…
• Grok — Concept · 1 mention
• Immortal jellyfish — Concept · 1 mention
• Immortality (effective biological immortality) — Concept · 1 mention

Places & events

• Memphis — Place · 1 mention · first at 25:25
  Building that giant data center, the Colossus or whatever it is. Yeah, the Memphis one. I think it is. That's where it is. Yeah, it's massive. Bigger than I think anything else…
• Factories — Place · 1 mention · first at 49:05
  Of and excrete some sort of a bofuel that we can use to power, factories and cars and whatever. That would be incredible. So, they're talking about it. They're working towards it.

Full transcript with timestamps

0:00 So it's not like we're just x amount of times smarter than an ant. It's like a different brain architecture that allows for completely new abilities. So then you can imagine there might be a step after humans and a step after that. They're not just smarter but they're like it's a step function. This is the end of the sort of biological range of intelligence. And then up here that's artificial super intelligence. We can't even perceive what's on the next level. Like as as ants are to us, if that same distance we travel up that ladder, we can't even imagine that. And the biggest illustration for me is like alpha fold.

0:37 And then it started spitting out predictions for other proteins that we we haven't sort of discovered or knew the shape of. There's some sort of a pattern that we can't even understand or predict. We feed into this neural net. And it's like, oh yeah, obviously there's this pattern to how biological life is formed with the building blocks. Obviously, like here's here's what I can make one up. Here you go. We had a good run in terms of being the greatest the apex intelligence in the planet. We've been the smartest things on the planet for a long long time. Hi, I'm Dylan Curious from the YouTube channel Dylan Curious and I am really excited to talk with you Wes. I'm very glad to have you here. Let's get started. I would love to just start with a question of like where when did you have a defining moment where you thought that AI might kind of be worth all this effort? So for most people, I feel like the Chad GBT moment

1:22 was the start of something, right? For me, I saw Elon Musk tweet. Chad GBT is getting scary good or something like that. I was like, what is this Chad GBT thing? So, I I went I checked it out and it was it was okay. It definitely had something to it. It wasn't magical. It wasn't mind-blowing, but there was some something there. I was like, "All right, this is this is interesting. All right, whatever." Um, next I started learning more about OpenAI and they had this really interesting hideandsek reinforcement learning kind of like a video game where they're training these AI agents to play hideand seek. And you know, it wasn't anything new. It wasn't anything mind-blowing. I think they just did a really good job of presenting it um in a very engaging way. They did a great job of showing the data and illustrating the concepts.

2:14 And it had these little blue and red people running it around with smiles in their face. So it was like very like visually kind of uh fun to to to see. And the premise was very simple. So there was two teams, two players on each team. Ones were one team was the hiders, one were the Seekers. And basically the Seekers, you remember that? Yeah. Um it's from a while ago. So this is before Chad GBT. This was before LM really even took off. This was a number of years before that I believe and it was basic reinforcement learning which has been around for a while. And so the red team would get plus points if they could keep the blue you know the the hiders in their visual field and the hiders would get you know a virtual high five reinforcement learning type of thing if they were able to stay out of the visual field of the seekers. Very very basic.

3:05 And so the first thousands of simulations or hundreds of thousands that they were playing, the AI agents couldn't do anything. They were just basically mashing buttons randomly. So they were like spinning in circles kind of like having these like jerky motions like no sign of intelligence or int or just just anything. And that kind of like I kind of I I was aware of this but kind of seeing it live um was very interesting. And in fact understanding that kind of ties into a lot of the stuff that's happening now because we didn't start them with any sort of human data if that makes sense. Do you know what I mean? We didn't tell them this is how I would hide. They were just like no you figure it out. We're not going to tell you anything. We're just going to punish you if you do it wrong and we're going to give you high five if you did it right.

3:54 And over time, over millions and millions of iterations, you saw intelligence emerge. And going through it, I was like, "Oh, okay. This is I mean, this is learning." Like to me, I was like, "This is what we would call learning. There's there's no way around it. This is machine learning." Um, and that was kind of the second kind of piece of the puzzle that kind of like opened it up for I was like, "All right, no, this is there was something resembling the human brain gathering information and this and that." And by a billion iterations, they found a way to they found some glitch in the system that the developers were not aware of.

4:34 So, they were able to like launch themselves in the air to land where the the se the hiders were and and it that was a surprise to the developers. Yeah, it's it's creativity and it's like almost like you know the whole idea of like we we created their little environment. We thought we knew everything about that environment but they figured out something where the developers like wait you did what? Uh I I just thought that was incredibly in incredibly interesting and then so so it's like Chad GPT it was that and then shortly after a few weeks after I think after Chad GPT came out I think three or four months later you had GPT4 and that's when uh Microsoft published the paper sparks of AGI. So they were saying this is kind of a prototype AGI. Reading that paper that was like the final whatever you want to call it nail in the coffin where I was like okay okay this is a thing that will attempt to do whatever you

5:30 throw at it and maybe it'll do well and maybe it won't just like a human being like might fail might but it will attempt it. It will it would understand what you're trying to say. And so between the reinforcement learning side of it and kind of the more general aspects of the LLM's GPT4, somewhere between those two moments, it just like dawned on me, okay, I'm like, we're on to something big. And I I I didn't know where it was going, but I was like, there's no chance that this is not going to be a big big big deal. And I immediately went all in on it. Yeah. You know, like that's so interesting because yeah, I can pinpoint kind of two kind of like a camel. There was two humps also where I first thought, okay, like there's something different between programming something and learning something. For me, that first moment was when I was watching Watson. This was a long time ago. And it was learning the

6:23 different ways you could write the characters like A, B, or C. And it was just learning not the averages, but something general about what the letter A is and the letter B is. simulated environment started making me realize like, oh, this could be like walking or reading MRIs or all these other things that didn't seem like it was a narrow domain. And then you have that second shock moment where you're like, oh, this is like really close to what it feels like might be happening up here. Definitely. And yeah, I like the fact that you said so it's not programming, it's it's something else. It's learning.

6:55 That was a big deal when I was like, wait, this isn't code. This isn't somebody writing out scripts. This is something again. Yeah, like the developer doesn't know how it works either. They're like, I fed in the data and then I tune the model and now it's solved, but it's not programmed. Like there's no way to go trace that all the way back. Absolutely. That was like a big mind shift for me. It was just like, okay, this is different. Yeah. For how many people actually still use Chat GBT? I suspect most of them think of it as a program. And we're kind of at a disadvantage trying to explain that, too, because you pull them both up on your iPhone, but one's an app and one's chat GPT, and they just feel so similar.

7:31 It's hard to get your head around the fact that behind the scenes, one learned everything it's doing, and one was programmed to do everything it's doing. Absolutely. And that's definitely one of the things that I think mo more people need to realize because also they're comparing it to um things. They're a script and they're like, "Oh, it's not as good as this program." And it's like, yeah, but one is static and one is like learning and it's completely different. Yeah, someone was like complaining that it's like it can barely do the square root of some big number. And I was like, yeah, but it also can like pass the medical exam and like write short stories which your calculator can't do.

8:02 So like it's a little bit impressive, you know, if you think about it. Absolutely. So are you kind of surprised that it felt like humans were so on this planet as the center of the universe and then we're like, okay, maybe we're not. And then you think intelligence is like the thing that makes us special because you see technology take away all these other things. And now it's like we're grappling with the idea that maybe even intelligence isn't that special, right? Do you feel smaller and smaller and smaller over time as you dig into this stuff? Yes, but it's good. Um I think when Ilascover like a years years ago he said if the only thing you value about humans is intelligence, their intelligence level, you're going to have a bad time. And that's literally what what I think he meant. And I think that's what we're seeing because anytime AI intersects within somebody's realm where they they feel very proud of

8:53 themselves, they're like, "Oh, I'm so good at this thing." And AI comes out and just like levels them, people get angry. And I see this like in in comments on my my videos, other people videos on Reddit, whatever. People are very scared of that. There's a lot of anger because they're they feel threatened that this thing is better than them. if they if they're very proud of their intelligence and their abilities and this thing comes out it's smarter. If that's the only thing you value, you feel like you're under attack. Um, and so that's what I mean by like if you let go of that a little bit, if you let go of that I don't know, ego, I don't know if that's the right word, but like you you just can't take yourself too seriously. Like yeah, we've been we had a good run in terms of being the great the apex intelligence on the planet.

9:38 We've been the smartest things on the planet for a long long time. Um, and now it's this emergence. Yeah. It's so weird. We're going to give that up to something we created, which is so different. Like, usually evolution came up with something that became the new apex predator. We might be the first ones to really just build something that we then succumb to. With succumb, you mean it's going to kind of supersede us? Not necessarily. You could kill all. Well, the weirdest thing is I guess succumb like in the sense that like even Okay, like let's imagine it's like super benevolent and we live in like the perfect zoo. it's still kind of succumb to like it is now in charge of the direction of humanity I guess you know and um it's just strange I mean maybe bad or good but yeah I mean I I don't know if that's going to be as big of a deal as people think it is and I might be wrong on this but I mean I'd love to hear yeah I mean

10:30 like calculators are better than us at calculating stuff like nobody has existential dread over that the chess AIs are far far superior to humans People still play chess. People still enjoy playing chess. There's this guy Scott Arson who is just really smart top tier physicist. He worked for Google on their quantum chip. Scott Aronson. Yeah, I'm pretty sure that's that's it. And Ilia Scog invited him to Moonlight for the OpenAI safety team um a number of years ago. So he which was not his field, right? So um Scott Aronson's field he's a physicist again extremely smart but AI was a brand new thing to him. So he came into it not as an expert but just as a brilliant person from whom they wanted him to contribute in whatever ways possible.

11:21 And so it was interesting to see this really smart person start fresh in a brand new industry brand new field like AI and grapple with some of these concepts. And some of the stuff that he came up with was brilliant. There was a talk that he gave. It was a TED talk eventually, but before that there's another talk on YouTube where he's practicing in front of a room and he's a lot more like relaxed and natural and it's it's a great it's a great watch. But towards the end people are asking him like so what happens when these AIs are better than us and everything. He's like it's not going to matter because like you know we have cars that are faster than human runners.

11:57 We still watch the Olympics. We have chess that's better than this. We still watch the Olympics. It's like or we we still play chess. So he was saying that we're going to compete with each other and we're going to get enjoyment from that. We're going to build we're going to write books because we're going to enjoy writing books and other people are going to enjoy reading other people's books. Right? If we're going to go for a hike, we're going to enjoy the hike because of the experience that it gives us. So, but what he's basically saying that instead of like everything being measured in economic units or like us trying to climb some social ladder, it's just we're going to be doing it for the experience the you know what I mean?

12:36 Yeah. Yeah. That's okay. So, cuz in my head I'm just playing it through like one by one. I'm like, okay, so we invent chess that's better than us, but we still play each other for fun and some people learn from it. And then maybe it drives cars better. And then it starts getting into more like decision making things. And this is where it gets a little fuzzy for me. Like I can imagine a government a president AI that does a better job than any human that we vote for. So we kind of put it in charge. And then I think about like medicine or mechanics kind of being outsourced until there's really nothing in my life of utility or of kind of use. But then I I guess there would still be like I'd get with my friends to like learn about medicine even though like I don't need to. or like would we just kind of hike and and do just pure physical things and then what if a robot's like better at hugging you and like loving you and like

13:29 then people are like well it's really hard to get another person to act the way I want and I can just date this thing then it doesn't it kind of feel like we've I don't know the chess analogy breaks down a little bit. Yeah, I totally get that. And this is going to be very interesting to see kind of what happens because on one hand, yeah, like right now, a lot of the things that we do, they're kind of economically driven, right? Uh we make decisions and people that make good decisions get rewarded for it, like in in in society, like if you're running a company or whatever. Um okay, so let's take let's say you take that away. Um and now you just do whatever you want. Does that make for a good life? I mean, we see people that are born into wealth that are miserable or people that achieve a lot of wealth that are miserable. I mean, like like Jim Carrey talked about, he used to talk about a lot how he like strived so hard

14:18 to make money and then he made all this money and then he's now saying like it's all meaningless and like don't even bother make doing all that stuff because it's meaningless. Like okay, let's imagine for just hypothetical like let's say there's a digital West Roth twin, right? like you have this iPhone app and it's just pretty much almost every time it makes a decision for you, sets up an email, like makes a restaurant order, like it's pretty much always what you think you would have done. Is that the kind of thing you would also outsource your vote to? Like would you say like why don't you go look at all the different candidates and like vote for me? Is that the kind of thing that you would say what like I want to impress this girl like you come up with a fun joke and then I'll say it. Like would you go that far or would you just truly say like I'm not going to open the app.

15:03 I'm not going to talk to my digital twin. I'd like to just do this on my own. I mean that's very thoughtprovoking. Um and I talked about this a little bit on a live stream. Somebody hit me with a similar question. I was like oh man. Cuz the thing is I it does feel so so the the idea is like what if like all the decisions were made for you, everything was nice and comfortable. You didn't have to stress out like you had everything. whatever joke you want to tell a girl like you had the best joke right there and there. Um, and we see a little bit of that in modern society where we have more and more stuff and people are not necessarily happy. In fact, a lot of people are miserable. So, I think you know what I you know what I think okay I think they nailed it in mat in the original Matrix movies. Remember when um Agent Anderson is is that his name or agent Smith? Agent Smith. Yeah. the um he goes, "You know, at first we created

15:56 the perfect utopia for you humans and you guys were miserable. So, we just scrapped that and we created this crappy world and you guys were just happy. You needed to be miserable." So, I think part of the we do need to be a little bit miserable that there needs to be some sort of a negative negativity or some sort of um something that we're striving against running. You know, we we all got to work. It's like certain dog breeds. If they don't work, if they don't feel like they're contributing, they're not happy. I think humans have a little bit of that. So, I think that at some point we're going to eventually AI or no AI, at some point we're going to reach through just automation or whatever. We're going to reach a point where no one's going to need to work as hard and we're going to need to and we're going to be a little bit more miserable. We're going to need to figure out meaning that's outside of all of

16:41 that. So, I I if I understand your question correctly, it's like would you I don't think we're going to be happy if everything is handled for us. I I don't What's the solution to that? I I don't know. Yeah. I mean, because it's such a human problem because Yeah. Like, you know, Yeah. you know, you don't want to fail right now, so you want to lean on the cheat code and then you lean on the cheat code month after month after month and your life has no satisfaction. You know, it always feels like the wrong move in the moment and it always ends up being the wrong move in the long run.

17:14 In a video game analogy, you're playing a game, you're interested, but you're struggling. you type in a cheat code and you go from struggling to it's effortless and you lose all interest. It's like life is probably the same. Like if you have everything, you're going to lose all interest. So it seems to me like building in some sort of artificial scarcity or artificial strife, hardship. That seems to be a good thing for humans. So at some point we're going to be thinking about how how to do that. I don't know. Like for me right now, if I'm not feeling good, what helps sometimes is I go and I have like a hard workout or I go running. I do something that's not necessarily pleasant, you know what I mean? It's something that's hard and that surprisingly I feel a lot better afterwards because it's sort of like that contrast. So if you if there's no contrast in life, you lose the ability to really appreciate the good stuff. So

18:05 I think that's going to be a big question. It's like how do we create contrast? Yeah. And if we have Yeah. We have 8 billion people on the planet and most of them need to now learn that that skill which is like the self-awareness that like I don't want the easy answer even though it's clearly all around me. It's like we have to get used to the Star Trek replicator world and not just leaning on it. You can't just get addicted to the holiday, you know? You have to somehow know your own weaknesses and like put some artificial fences around yourself to make it harder. So yeah. Like, do you feel like we're going to regret inventing AI? Um, or do you think this is the same kind of progress of technology lifting up the the world and a better world? To me, you know, each big technology from the past kind of what it gave us is leverage to do more. And that of course usually means we can do a lot more harm. We're a lot

18:55 more good. AI is the biggest leverage technology we've ever had and potentially will will ever have. So yeah, I think it's going to be the the you know, the day we invented AI is going to be like the day we rule forever and just like why do we do it? Or it's going to be the greatest day for humanity ever and it's going to be one or the other. You know what I mean? Uh and I don't see how it could be neutral and do you know what I mean? Like it's going to have a big impact, positive or negative. Like what would you put your P Doom at? I don't like the idea of just talking about P Doom because I think there's a number of other really bad dystopian outcomes that has to be considered. I feel like if we're just talking about P Doom and that existential risk, some sort of catastro catastrophic X risk, we're ignoring like for example somebody joking and I said, well, okay, we have our P Doom. What's

19:49 what's what's P 1984? So like what's the chance of uh you know that book? Yeah. Yeah. What if like some tyrannical regime takes over? Everybody's surveiled all the time. Everybody's brainwashed and there's no way to go against it or or overthrow it because I mean if you're talking about P Doom, a lot of the solutions to not pursuing AI development is kind of more centralization, more cooperation. There was some talks about maybe even some sort of a like a world government organization that's like top down making sure that nobody's developing AI. Um I mean if you're really concerned if you have a high p doom what is your only solution? It's really like get the governments involved and like shut down development right if you if your pdoom is 99%. Like what else what else do you do? But then what happens if you know the government decides to abuse that or develop it but for their own needs. So it's like I

20:48 think we need to talk about all the risks holistically. That's why like Daario Ammedday, how he kind of approaches because he's like, "Hey, yeah, we got to talk about X-Risk. We got to talk about P Doom. We also got to talk about what happens if an authoritarian regime gets access to this technology before everybody else." So he's like, there are multiple bad things that can happen. We have to look at them holistically, if that makes sense. And of course, if you centralize everything, you increase the the risk of an authoritarian regime or whatever potentially, right? like people with high AP dooms like Eleazar Yadkowski, they yeah like they kind of you start asking like what does this mean? Well, it means governments get more kind of like tightknit about what's going on and if people are not some countries are not compliant at a certain point you know if they're really not compliant like the

21:33 same way with a nuclear you know checking program like you might have to you know bomb a server or center or something and like stop and like show that you're very serious about them participating or cooperating you know. We're talking world government with complete control over countries, something in that direction. And there was a speech where um somebody was actually kind of suggesting that talking about that. So it's not me just making stuff up. I mean that's how you would deal with that. That also scares me. You know what I mean? Right. Yeah. No, that come with a a much bigger sense. I'm kind of curious just out of like the the people that you've sort of seen like the Elon Musks, the Sam Alman's, the Demisacuses of the of the universe, like who are you kind of leaning towards like agreeing with like who do you sort of trust the most and sort of not trust the most or like how you feeling about the

22:21 leaders in the AI space or the CEOs of the major tech companies? I feel like first of all, I try not to fall into the thing where anybody's like an angel or a demon. They're all good or evil. I don't think anyone's like that. Everybody's like flawed mix of whatever. So, I don't think everybody's somewhere in in in the middle. Um, in terms of trust, the only person like I feel like I fully trust is is Dennis. I don't know why. You know, he I actually feel the same way. Yeah. He was just there before everyone else and he tried to like he wasn't in favor of pushing this into product so quickly and he seems like he's genuinely smart and he's actually there on the ground level building stuff. He understands the e technology and it's weird because when you do interviews with him like no matter what they throw at him like you want to talk biology or this he he knows everything and he's got like a just a really great grasp on

23:14 everything and he genuinely seems like he cares not about himself or whatever but humanity. Everybody else I don't think they're necessarily bad but they're more more human. So you know what I mean? Like just like the rest of us they have their own ambitious ambitions and goals and whatever. I mean, do you think Sam's pushing product too fast or do you feel like his take is incrementally pushing stuff out allows humanity to kind of like absorb it and figure out what to do with it? Um, that kind of after like watching what's been happening that kind of is making sense to me. uh the idea that let's just iterate and push it out, iterate and push it out and then we're able to sort of like on the fly build certain guard rails and figure out where it's going to start breaking stuff. I know I'm sure a lot of people think that's crazy. They think that's dangerous. I'm I'm not saying it's safe either. I'm sure at some point they're

24:10 going to push a model that's going to freaking break something. Uh meaning that it's going to cause some damage, cause some issues. But maybe that's how that might be the safest way. It does seem like it has a little bit of an evolutionary or marketplace dynamic to its kind of safety like in the same way the cyber security is kind of constantly evolving with cyber attack. Exactly. And so I think I think there's some truth to that. Now of course that might play into you know his hands in terms of like the company valuation and stuff like that.

24:42 So it might be he he's might be playing his own game, but just on the surface it makes sense this idea of rapidly pushing stuff out there and just letting societ forcing society to figure out how to deal with it. I know that sounds bizarre, that sounds risky, but at this point it might not be such a bad idea. It's like a very evolutionary process like like you said. I think that's a good way of looking at it. And then Elon Musk went from just assuming that everything's, you know, AI is going to kill everything to now I don't know if he's more positive about it or at least he wants to at least control it to be part of the development. Um I'm very much looking forward to because they're building that giant data center, the Colossus or whatever it is. Yeah, the Memphis one. I think it is. That's where it is. Yeah, it's massive. Bigger than I think anything else as a single data center. So the next iteration of Grock,

25:34 if they build it with the full power of that thing, that's that thing's going to be massive. You feel like scaling hasn't hit a a roof. I never quite know. Like I hear people say like, "Oh yeah, we've kind of now it's an architectural problem." Sometimes it's just a scale problem. But the thing is like scaling is an exponential cost to like a linear progression, right? So you 10x the thing, you get, you know, you double the the sort of the the abilities of the model. And certainly the jump from, you know, 4.0 0 to 4.5 wasn't super awesome and impressive. But here's the thing, we also have found a lot of other ways to scale, not just more compute, you know, with with test time computes and a lot of the different sort of architecture changes and stuff like that. So, I feel like we're finding other avenues of improving it that are also scaling it, but it's probably not scaling in the same like not just more more compute or

26:24 whatever. So, I think that's not going to be like the only path to improving it. There's going to be a number of different paths and I think that still just we're going to need more Nvidia chips and more, you know, stuff cranking out data and stuff like that. Like when you look back, is there a favorite uh video that you put out there that you're like most proud of? The videos that tend to do poorly are some of my favorites. The ones where I just go deep into some rabbit hole and it's just kind of there for me. Uh but it's not like what appeals to most people cuz new model releases and stuff like that, that's what people want to see. The stuff that I mess around with is like for example there's one that showed that models them AI models have their own mental models that they understand how the world is even parts of it that we don't kind of tell them about they kind of build an abstract representation they

27:14 guess at the existence of the outside world that was one of my favorite ones actually is that new is that the anthropic paper this was actually one of the older ones and that's the one where Andrew A he was saying these models understand because So, uh, Oll GPT, I would look at that cuz that one is mind-blowing. So, in Athell GPT, they fed him games from, um, a game called Athell and just the notation of the moves made. And after a while, the model started to be able to predict what move comes next, which is interesting. And that's kind of like what they're supposed to do. But later on once they started digging deeper they realized that the model had kind of a representation of what the board would look like. Nobody told it about a board.

27:58 Nobody told it about players. Nobody told it about squares. But it it it created a mental model of what the board looked like. See like what if that's how consciousness builds up, you know? Like you think about yourself when you were like 3 or four years old. You just didn't have so much buildup. So it almost felt less. Yeah, dude. That's crazy. It's weird to think about. So, I've only like kind of skimmed somebody who summarized it, but I know there's an anthropic paper from this week where they were using those autoenccoders to try to look into how some questions were answered. And when they asked it to do a math uh edition when they asked it through theory of mind, how did you come to the solution? It said, I carried the one and did all this stuff. But then when they watched it through the autoenccoder, it actually had like a whole network that activated about like what are ridiculous numbers and it kind

28:40 of honed in on what the average answer like zero wouldn't be a good answer. A billion wouldn't be a good answer. And then another part of the network kind of did a different thing than it thought it did to calculate it. Like it it did add the numbers but only in the first column and then just generalized on the other half. And it was like it's so fascinating. It's it's so bizarre. It's so fascinating. I can't wait to discover more about this stuff because that's like a whole rabbit hole of like what is happening those neural nets because it's not like in the beginning people like oh yeah just stocastic parrot predicts blah blah no way there's some insane in insanity happening there and yeah and I'm going to ask you a question that almost by definition doesn't have an answer so we'll see how you handle this but when you think about artificial super intelligence when you think about an intelligence that's magnitudes and

29:27 even architecturally ally different and more powerful than what the human mind is. How do you get your head around it? Before even chat GPT and LLMs, there was a uh blog post on wait but why and it was called there was AI revolution parts one and two. He was explaining what intelligence is. So we kind of think of intelligence as this line that just kind of goes up and the higher you are the smarter you are. But he also said you also have to think of it as almost like this a kind of step function to where different brains are not capable of understanding high level stuff. So like an ant might an insect might be here and then like a chicken and a monkey and then a human. So it's not like we're just x amount of times smarter than an ant. It's like a a different brain architecture that allows for completely new abilities and stuff like that. So then you can imagine there might be a step after humans and a step after that.

30:24 That's that's they're they're not just smarter, but they're like it's a step function. It's a step above. And so and then he shows this little he kind of zooms out in the picture and he's like now imagine the staircase going up up up up up up. And you know we're like over here and way up here is he points to and says okay this is the end of the sort of biological range of intelligence. So this is where she, you know, organisms, this is as smart as they can get. Let's imagine. And then up here, that's artificial super intelligence. And that image kind of blew my mind because I'm like, what is that even like me? It's not just a million times smarter. It's a hundred steps on that sort of a ladder of intelligence more advanced. Like we can't even perceive what's on the next level. Like we can't even imagine what is one level above humans in terms of like like as as ants are to us if if if that same

31:24 distance we travel up that ladder like what is that like we can't even imagine that so you think even if I imagine like a thousand Einsteins in a room who all answer and work together instantly that's not even right like it's a whole qualitative architectural change in intelligence that is even beyond speeding things up the known like the known way we solve problems. I Yes. I really don't think that it's just like this a linear thing where it's just like 100x a thousand x a million times. It that's not representative of the entire thing of of what intelligence is. And the biggest illustration for me is like alpha fold because we knew how certain proteins folded. And it's this highly complex thing that is like impossible to brute force cuz the number of variations that it can fold into is like more than the number of atoms in the known universe, right? So it's like this like insane number. So you can't brute force it. So,

32:22 we slowly, meticulously figured out, I don't know, like 200,000 3D shapes of these proteins. We fed it into AlphaFold and then it started spitting out predictions for other proteins that we we haven't sort of discovered or knew the shape of. Like, what does that even mean? That means that right there's some sort of a pattern that we can't even understand or predict with with supercomputers with anything that we can code with anything up until now. It's not we we we can't do it nor can we even understand what that pattern might even be. we feed into this neural net which it's not old at this point but you know it's a few years back and it's like oh yeah obviously there's this pattern to how biological life is formed with the building blocks you know the pro obviously like here's here's what I can make one up here you go and it's like so um I guess what I'm trying to say is like you know it's like those

33:21 intelligent tests where they have they show you shape they rotate it rotated rotated if you're smart enough you know what comes next so these AI and neural nets, they're able to do those intelligence tests that we can't even like I don't think I think if you took any human being you made them a thousand times smarter or like you said take you know a thousand Einstein Einsteins or whatever. Yeah. Like they're not going to crack that problem. That's so interesting. Like I can so I can sort of expand my thinking to say like okay if I was born with a tail would I be able to control it the same way I can with my arms? Like probably the brain seems to be able to hand if I had the kind of sense of a smell that a dog has, could my brain smell that well and handle it? Probably. But yeah, it does feel like if you were to take my brain and it was in say like a vehicle, would it actually have figured out how

34:09 to move tires and steering wheels, like maybe it's the wrong architecture? It's like a biological thing that doesn't fit there. Maybe it could. But then as you get further and further away and think about something that's only learning data about the shape of a protein, it lives in not the 3D world. It doesn't wake up and and have the same sense we do like it's still super intelligent but it's so different from what we usually think of. Not with supercomputers cuz supercomputers so what? Like it's not about brute forcing it because again there's more variations there number of atoms in the known universe. Like you're not going to brute force that thing.

34:43 there's something else that it's doing that we can't even I don't know approach. I I mean are you comforted by the thought that like there could be a superhuman intelligence out there that for all the things that humans worry about have solved like maybe no natural disaster will ever hurt you. Maybe no person, you know, can like break the law or something because this thing can predict it. Or do you just see that as like true like terrifying because God knows what it's going to do and what that means for your future and unknown is even more stressful. I I feel like whenever we talk about intelligence and throughout history of how we write about science fiction, we it always flatlines.

35:23 So in other words, if you think about okay, intelligence gets to this point and then it stays there and fixes all all our problems, but it it's seeming like now it's like exponential. So it just grows grows and grows and grows and grows and we don't know if there's some limit. I don't even know what that means because again all these situations where it like you know it's like oh it solves a lot of the problems and it creates order and scientific progress okay but then like a day later it's a thousand times smarter and a day later it's a million time like what is that even like yeah you're right you're absolutely right like for one instant it could be like I'm completely aligned like I'm going to take care of earth and then it's like 50,000 times smarter than it was when it was already superhuman smart and And we're like, is that do you still have goals? Like, do you still even recognize us? Like, yeah. Like, what is

36:13 that even like I I'm I have a really hard time like even understanding what an exponentially increasing intelligence looks like. Now if we if we talk about more from like how it's in science fiction where like both um Elon Musk Demi Sasabis they both point to the culture series those series of science fiction books as what they see a benevolent um um artificial super intelligence kind of that's what they're hoping for that's what they're seeing great series of books uh for people that haven't tried it I would do the player of games that's probably one of my favorite great Great one to get into.

36:51 Very interesting. But even there, they're projecting, you know, whatever thousands of years in the future. But even there, they present this artificial super intelligence as it flatlined. So, it's not exponentially getting better, better, and better. It's like it's this smart and it's able to create a great life for humans. Humans live a long time. We have all these amazing abilities. We're able to augment ourselves in all these spectacular ways. Um, we're terraforming planets. We're building incredible stuff in space.

37:22 We're finding other civilizations. And if those civilizations aren't sort of as enlightened as us, we go there and we kind of like help them figure stuff out. You know what I mean? So, it's this massive benevolent thing, but it's still static. It's like super smart and it stays at that level and we don't Wow. It almost defi because it almost defies storytelling narratives, right? Like how do you build a story about an exponential exponentially growing intelligence, right? Yeah. The movie would just Yeah. It would change every few seconds and then be over. Yeah. So that's the thing is I almost feel like it would make more sense for us if there was some limit to intelligence. If that makes sense. If that makes sense. Like it gets to a certain point but then for whatever reason maybe maybe intelligence can only go so high. Maybe there's not some, you know, if you're measuring in IQ points or whatever, you know,

38:12 whatever, you know, maybe it can't get to some astronomical numbers. Maybe it gets to, you know, a million, right? It gets to sort of the intelligence of the collective humanity's intelligence and can't go beyond that for whatever reason. Maybe it's like a benchmark that gets saturated, right? You know what I mean? So, there might be some limit that we're not aware of. If there's no limit and it grows exponentially, I I my mind can't grasp that. Sometimes I look up into space and I see like nothing out there and I'm like, where are all the aliens? And then I see this AI on our planet growing and I think okay, if it becomes super intelligent, then maybe it'll say like what's the point of and maybe we're augmented with a neural link type thing and we're like, well, what's the point of like this biology? It's kind of weak. It's flimsy. It doesn't allow me to go into space so I'll become a robot. And then your consciousness is

39:02 kind of like what's the point of being a robot when I can just like shoot across I can shoot information across the universe at light speed so why don't I just become some kind of a signal and then like do you ever think that could be the like the firmy paradox or like does it seem like maybe the reason no one's out there is cuz no one stays at this stage very long and whatever it is we don't really like see it or understand it. So I mean to me the yeah the firmy paradox is very interesting. So it's like why haven't we contacted anything out there? The scariest outcome is kind of like the dark forest hypothesis is that like just everyone out there figured out that you got to stay really quiet or something horrible happens to you. Like all the other aliens, they're like, "Don't send any any radio signals. Don't send anything out there. Just stay quiet." And then here we are like broadcasting. I think

39:46 we sent Elvis music into space or something like that, right? And they're like, "Oh, they're about to get cooked. Just just watch. They're going to get cuz whatever. There's something out there that's bad." That's the dark forest hypothesis. That's the scary part that we're just like broadcasting and that might be really really foolish. Um, but what you're saying, yeah, it makes a lot of sense. It's like what if every civilization, so if I understand correct correctly, you're saying every civilization advances on its own planet up until a certain point.

40:15 I'm imagining that it just evolves past anything that we recognize right now as like robots or biology. So it doesn't get to the point where it needs faster than light travel because it gets to some sort of a enlightenment upload itself to the cloud and become gods before it even need. It's like why do you need rocket ships if you have this if you're just this pure energy? Is that right? That kind of right is am I guess I mean like I said it's sort of vague like sometimes my mind goes into something like a black mirror like well then you just build on a planet some kind of computer and just you live in the computer because it simulates it.

40:48 Other times they like more like Soma the computers are kind of like out there with people on them but they're floating around in space. And then sometimes it's like maybe there's a way to communicate with through signals that is just maybe something quantum or something that we like aren't even really ready to even explain yet. And that's kind of the best place to make a information network where a graph can actually connect all sorts of things together in the universe and we're just not tapping into it yet or something, you know. No, I I think that's totally that makes a lot of sense to me because it's like that thing that like some people have mentioned it's like you know we always thought we we all thought that we would have uh flying cars but instead we have 150 characters or whatever right so the the digital thing evolves so much faster than the physical like what if we have super intelligence it something happens that

41:38 makes traveling across light years it's like why like what if yeah there's something so much more amazing out there that's more digital or energy based or what you're saying like I get it that it's like a little bit vague in terms of like we don't know but like what if nobody gets that point because it's like why there's something so much bigger and better on the technology tree before you know faster than light travel that just negates it. I that makes a lot of sense to me like when people say AGI like what does that definition sort of feel like to you? What are your favorite ways to define it? which which ways do you wish people defined it? So Ethan Malik, he made this concept that I think more more people should be aware of. He he described it as the jagged frontier of intelligence or the jagged frontier of abilities. And so basically the idea is if you think about um what we think of

42:32 as difficult or hard as humans, it's kind of on this curve, right? So if you can tie your shoes, you can probably like brush your teeth. If you can put up a shelf, you could probably fix a bike tire. And so like certain things for us are on a like we think of certain class of tasks as being the same difficulty level. Uh for AI systems that's not that curve is not the same. It's like this jagged thing that's like it could be superhuman PhD level in one aspect and it can be like can't make you a sandwich or something. Yeah. Or like the whole straw how many Rs are there strawberry or whatever. remember that whole thing where it's like it can't count, right?

43:11 It's like, well, a child can do this. And so, what I think a lot of people misunderstand is when they find the areas where AI is really bad at, they're like, oh, well, how smart can it be? It can't count, you know, the number of RS in a straw in strawberry or whatever. So, they discount its intelligence. But guess what? At the other side of that jagged curve, it's solving geometry problems and medical problems. I wonder if you saw the live stream with the the PhD professor that's working on um like some something with measuring black holes and he took a year to write his dissertation and he coded up this it took a year to write the code and then when 01 came out he told it to write the code and like like that within half an hour he had it and I remember watching that live stream and he's just sitting there like this like 2:00 in the morning he's just like you can see he's got like this existential kind of moment like

44:03 what like I spent one year like He for you know all his life learned how to do stuff like that then took a year to write and and it's just like boom like that. Are we like if it's able to do PhD level work but he can't count the number of letters like at what point are we like okay this is much smarter at humans in the areas that it's good at and the fact that it can't do this one little simple thing doesn't take away from its intelligence. Um, so I guess it's like how you're defining because if if we're saying like large language models in a lot of small well-defined tasks are far better than the average human already.

44:42 So by by that definition of AGI it's already AGI. We're there. Do you think we have AGI then? Oh no. I would say by that definition we do because it fits that definition. Open eyes definition is it's going to be doing the majority of all economically valuable labor. We're not anywhere near that by that definition. You know what I mean? So it seems like AI and biology is making strides every day. It seems like it's doing things with genetics and microbiomes and uh the vyro that would would have blown my mind like 5 years ago. And it's leading me to think that long lifespans are very possible. Um where do you fall on that? Do you think do you do you see yourself still alive in 400 years, 500 years? Like do you are you preparing for that or do you think that's unrealistic? I think it's it is realistic. So I kind of subscribe to the to the idea of um at some point us hitting this escape velocity where for

45:38 every year that we live our life expectancy goes up two years for example based on various breakthroughs and stuff like that. there was this person um Aubrey Deg Grrey and so his idea is basically and correct me if I'm if I'm wrong if I recall correctly so he's talking about negligible scinessence so the idea that eventually we can get to the point where we slow down aging to such a point that it's not even visible noticeable um and al talks about there's a couple different things that tends to kill most of us like cancer heart disease there's some plaque build up in the brain like these things that if we're able to fix those and then slow down aging. We would be effectively immortal. Uh we could still get hit by a bus, but aside from stuff like that, we could stick around forever, which I think we we know certain life forms like there's a form of some jellyfish that could be immortal cuz once it reaches

46:34 the end of its life cycle, it just sort of like becomes a baby again or something like that and it like just keeps going. So, it's not outside of the realm of possibility. And I feel like now that hopefully science is going to be greatly accelerated, scientific discovery is going to be accelerated with the AI, I feel that it's very possible that people like our age, like people are alive today will be part of that escape velocity. Um, and really what's interesting to me is we're entering a new kind of era now where before and this is I Jensen Huang talked about this, Dei Hasabis talked about this the idea that before in biosciences we had kind of the the era age of discovery like we found ingredients in nature that did this or we find found viruses or this and that and we're slowly entering kind of the bioengineering age where we're like constructing life Now we're beginning to build stuff. Um, you know, Deep Mind and

47:37 their spin-off isomorphic labs, they are, you know, they're building new proteins that like bind to other proteins so that we're able to control their functions based on whatever abilities we want them to have. That's m it's brand new. Like we're building life, right? Like if you think about it, it's not out of the realm of impossibility that in 20 years you might go for a hike and you might see like threeeyed frogs and like creatures that you've never imagined before because some agent out there spun one up for some other agent and like it just had its own natural course of like evolving.

48:15 I mean, do you can you imagine a world where we have all sorts of biological creatures, some smarter, some some less smart, like totally new form factor stuff that nature wouldn't have come up with? Yeah. I mean, the way you described all this time of like, oh, we need regulations on this, like you don't want to run into some threeeyed whatever on your hike. So, I mean, but yeah, for sure. You know, Google DeepMind has made it a mission. For example, they said one of the things that they want to do is to create bacteria. Basically, bacteria, we're able to design it to where it can eat whatever we want it to eat and excrete whatever we want it to excrete.

48:50 So, for example, a bacteria that eats plastics, we can put in the ocean to clean up all the pollution. That would be incredible, right? Uh or a bacteria that eats something that we have a lot of and excrete some sort of a bofuel that we can use to power, you know, factories and cars and whatever. That would be incredible. So, they're talking about it. They're working towards it. There's active efforts in that direction. So that to me is absolutely phenomenal. Like a just little tiny nano robot, you know, not robots, but actual biological life form that fixes pollution and the crap in the oceans and all of this stuff sounds incredible.

49:28 Maybe g maybe gan drives would be another place we could start like uh do you familiar with the idea of a gene drive like that it's in the the sperm and gamet cells or whatever so it continues through evolution, right? There's two two kind of interesting projects for the gene drive. One is with uh mosquitoes. And in theory, you can let a few of these mosquitoes out and they keep mating and they make every mate that the baby comes from have no ability to spread malaria and then quickly it's like 2 4 8 16 and you can kind of knock out the whole population fairly rapidly. And then the same for mice that have ticks and there's a potential that like ticks don't end up on the mice and ticks don't end up biting the humans. So in those cases it seems kind of good. But once you make a a change like that into an actual animal out in the world, there's just no there's no recovery from that, you know?

50:21 Yeah. Yeah. I feel like we're not at the point maybe we can start experimenting with that. Hopefully at some point we'll get to it. like we know we make this change, this is how it like once we have full visibility into it, we're going to be able to make better changes. I feel like right now we don't get the genome. We're like scratching the surface of what we know about it. What's your thoughts about AI predictions in terms of pre pre-rime? Um, are you like in favor of a a model run by a police department or some kind of authority that starts to hone law enforcement in on somebody who might commit a crime?

50:54 Yeah, I mean this is kind of like a weird rabbit hole because it's like obviously it can be abused and you know one of my fears is like what happens if tech like this falls into authoritarian regimes hands. Yeah. Like if you can predict if somebody's going to commit a crime, you can also kind of predict if they're going to vote for your party or not and it's just they're so close to each other and then you can be like well that that's a crime or you predict that they're going to commit that crime. Yeah. It just it's a slippery slope. So yes, I would be extremely careful about that. I guess is the do you see a way for democracy to continue in the age of um like hyper intelligent agents? Yeah, I guess maybe it would have to morph a little bit because on on one hand right now I mean yes it we have a democracy but also it's not like every single person gets asked what they want and then that gets like perfectly reflected.

51:45 It's still consideration of power etc. with AI like we each can talk to our own assistant and be like this is what I want and then it sort of being able to figure out how to um really have a true representation of what we want. Um also I feel like there's a difference between what people want and then how they want that to be accomplished. Right? So I think that you know most people if you ask them what do you want to happen they'll be like I want this this and this. But if you ask them what is it they generally what outcome do they want the kind of the means to that outcome and what how they they think we should handle it might not be the same and AI might be much better able to figure out how to get us what we want. So at that point we shouldn't be voting for the policies or whatever we should be like how do you want the world to look like? Yeah. Like I've thought about rank choice voting before and how

52:41 sometimes the best candidate is like neither of the two that end up at the top of the ticket. But how you get to a solution is so nuanced that we don't even think of that in terms of democracy cuz there's never been a way to really vote or like work towards a super complicated super shared goal. But maybe AI could do that. Also, some people are beginning to talk about this idea that, you know, with AI, you could potentially have kind of a centralized government, even though that kind of like uh um has failed before likely because it was just not like it's better to have a democracy and kind of everybody's like separated because you know, then everyone's making decisions that are contribute to the whole. Could a perfect like AGI just run everything and do it better than humans can? I can see that happening. So the question at that point is, do we allow it to take over? And so Dave Shapiro

53:35 asked his audience, I actually asked my audience, most people would be in favor of this as long as there were some safeguards, right, for for it not taking over. So most people would be like, what was the polling numbers? People are in favor of that. Uh, so so well, here's how how I phrase the question. Yeah. Yeah. I'm curious. Because I I I said, let's assume this. assume you looked at this proposition and you were reasonably sure that it wasn't going to be abused that there was not going to be any sort of nefarious stuff like you were re that the the AI was aligned. So my point was assuming that there was no catastrophic things that could come from this. So that was the assumption. Okay. Would we use this to run the government and make it more efficient and blah blah blah?

54:18 Most people said yes. I forget the numbers, but um it wasn't like completely one-sided, but maybe like 6030 or something like that. I would have to look it up, but most people be like, "Yeah, sure." Like make it more efficient, like you know. So, you know, uh what do you think about this? What if you have on your phone, like on local hardware, not connected to the internet, a digital twin, and it knows what you're going to do pretty well. Would you want it to go out there and read like an actual uh like say like the bills that pass on the Senate floor that no human really reads except a few people? Like would you want it to go read that whole thing and then vote constantly on uh on your behalf? It's an interesting concept and might be it it seems like it would be a good concept. Um I I feel like a lot of the stuff with with voting and stuff like that is because there's a certain mis mistrust. So the reason we

55:10 we vote is because we want to make sure that if there's any abuses, we can kind of vote the power away from those people. Assuming there's trust and we're all sort of aligned, we're all on the same page. It might make sense to take the smartest, most effective person that's capable of bringing about those changes and be like, you do it. Assuming we trust them 100%. Which hasn't really existed before. I mean, a lot of the stuff I feel like is like just making sure that no sociopath climbs to the top and starts messing with everybody else. Um, but yeah, I mean, the idea, but but let's say we can't have perfect trust.

55:49 Would the idea of myself and everybody else in the community doing that and and kind of summarizing it either to myself or just voting my on my behalf, assuming that it's really kind of a perfect digital twin? Sure. without something better than that. That seems like a reasonable Yeah, that seems like a reasonable thing. What So what about um the way you now think about your own thoughts? Like how has that evolved maybe over the last decade or so as you've kind of like learned more from this these early OpenAI um games into this kind of like full-time life that you now live in? It do you do you feel like your brain is similar to a neural network or do you still find yourself finding something special about it? Oh, that's a great question. I I feel like the more and more it just seems there's more and more parallels between neural networks and kind of like the stuff that happens in our brains. I think we have a

56:41 lot of other systems that are working in parallel like whatever like emotions are. I don't know. You know what I mean? That's that's something separate. It seems like some of the chemical process that's a little bit more something. Um but in terms of like the research that anthropic is doing, you know, they they they find cluster clusters of neurons that encode certain subjects or certain concepts, then you're able to like activate those and that model gets like a little bit more obsessed with that thing. I mean, that certainly seems like how the the human mind works and the neural connections to stuff like that.

57:21 Um so and then the idea of intuition, we have certain intuitions. So things that we know or feel but we can't quite express in words. You know that's similar to how the neural nets they might have certain thoughts or whatever that they have in the latent space. So they're thinking about something that we can't readily observe. Um or now there's this idea there saying like system one and system two thinking. So something where you really have to think about hard versus a little bit more like ingrained beneath the the surface faster reflexive thinking. I mean it really seems like we're just recreating what has evolved in our brains and our bodies. This seems like tons of parallels. And I also feel like at some point AI, the field of AI and machine learning is going to start feeding back into like I don't know the study of the brain, you know, in psychology and stuff like that. I I really do. It's going to

58:21 start like feeding back into it. I mean, do you get the sense that we have free will? Like where do you fall in that? Do you think I don't know every thought like this idea to come on like have a talk with me? it was just all sort of predetermined and could be calculated ahead of time or do you feel like you're got like actual decision- making somewhere in there? I mean, I definitely think we have free will. I just I don't even know how you would go about proving that or you know what I mean? Um because if you have free will, it's kind of also implying that you're going to be somewhat unpredictable to a to an AI.

58:51 Have you seen that social similacar paper out of Stanford where they took a bunch of Chad GPS they put in a village and it kind of simulated um living in a village and carrying stuff out? I think and there was a couple interviews with the researcher behind that paper and so he's saying in the future we're going to be using these systems to simulate for example if the government puts out for a new policy we can simulate how people react. I believe that these AI systems will predict extremely accurate how large groups of people will behave. So in other words it's going to be able to simulate them and predict what occurs.

59:27 So similar to you know how insurance works it's like these what is it the actuarial tables or whatever like they're statistically really accurate. It doesn't mean that they can predict when any given person will die but yeah there's big overall patterns. Yeah maybe big crowd control will be more controllable than individual predictions. I really think so. I think it's going to nail the the crowd large crowd you know population um predictions. Could it predict what a single person will do? I don't know because I mean if it could then obviously then that would imply we don't we don't have free will or some sort of lack of the ability to make our own decisions or Yeah. If it can predict that that means we're following subscript obviously. So what about ethics and morality? How um do you say please and thank you every time you use a large language model? Do you do it because you think that's what it wants

1:00:15 or do you do that because you want to do it? That's an interesting question. So, I I probably tend to use please and thank you more than I think is reasonable, if that makes sense. Like, why am I typing please? But but I do. And uh do I think it's because it's important? No. Um not logically. Do I think there might be a chance that it would be? I mean, I don't know. I try to keep an open mind. So, a lot of people like Jeffrey Hinton, lots of others, they're sort of suggesting that these things could be have some sort of consciousness or some sort of um potentially having good experiences or bad experiences, pain, pleasure. We simply don't know. We we we don't know.

1:00:58 And we got to be very like clear about that. We can all have like our like my gut feeling, my my I don't think that it's experiencing anything negative or positive. Am I sure? No. Oh, I have no way of knowing that. I mean, can can you can you kill an ant or a bug without like if you see spider, do you squish it without any second thought for insects? No judgment if you do. Like I've squished I kill spiders. Yeah. No, for sure. I I I kill insects if they're like whatever in the house or whatever. It's not zero zero cognitive load or whatever. There's a little bit like, okay, I'm killing something. Yeah. Like I'd prefer if it was outside and not in my house, but if it's in here, I got to kill it. Yeah. And I wouldn't go out of my way to do something like that, but yeah, you do it, but there's just a little like a little part of your brain that's like, well, this isn't great, if that makes sense. Yeah. Is that similar

1:01:46 to what you feel about their intelligence now? Are you kind of like in that same realm where it's like a bug that an ant that doesn't seem like it's close to a mammal yet? But so, not really. And so I guess let me just rewind just a little bit, take a step back because I think that with a lot of these words that we've used for so long like consciousness and reasoning and thinking as we're developing these AI like a lot of these words stop making as much sense. They we we haven't fully defined them. So consciousness I feel like there's two parts that we kind of lump together. One is just the idea of like some sort of an awareness awareness of existence.

1:02:28 And the other part is some sort of a sub subjective positive or negative experience. Right? So if we think of something as conscious, it might like something or it might not like something. It might have pain or pleasure or joy or sadness. And I think we got to separate those two things. Like can can something be can something have some sort of an experience without it ever being negative or positive? Just some sort of a neutral experience. then we can't hurt it. Right? I believe that that's very possible that these models either now or at some point in the future as they get more complicated they'll develop some sort of awareness.

1:03:09 But I think we immediately lump on this idea that oh if they have this awareness that means that we could hurt them and we don't want to hurt them. You know what I mean? they might have like a negative experience and that's where stuff gets kind of crazy a little bit um scary also probably because of all the Black Mirror episodes like you mentioned a lot of them kind of have that concept like what if we take this thing we create it thinks it's real and then we put it in a situation that it doesn't like um at this point it's really hard for me to accept the fact that something like that can have a positive or negative experience because we know where our positive or negative experiences come from right pain comes from the nervous system. Emotions come from some other system. Like if we're happy about something, that's like dopamine and serotonin. If we're sad. So in other words, like we can point to

1:03:54 some systems in our body that define if our experience is good or bad. For LM, they don't have that. So can it be conscious without ever feeling good or bad? I I believe yes. But if you if if some people are saying that means that they might feel pain or we could be torturing them somehow to me that's hard that's harder to accept that they would have to know like through what mechanism are we causing pain. Does that make sense? Yeah, it totally does because we're in this threedimensional world and we have dopamine that has been given to us for all these pleasure moments and then our brain learned from that signal.

1:04:31 So pulling that signal out or whatever is part of our whole mechanism which doesn't apply to something like claude running on servers. So it's not apples to apples to ask. All right. So I feel like there's only so much that the human brain can take about its own place in the universe and like the nature of intelligence and thinking this deeply about this many things. So maybe there's a good place to just end this first two-part podcast. In case you haven't already heard of Wes and you're seeing this for the first time, there's the first half of the podcast up on his main channel. You can go search that at Wes Roth. You can also subscribe because he does amazing video content all the time.

1:05:08 And if you're not doing it already, follow him also on X because like I get a I first learn about a lot of breaking news from your feed. So you're really on top of it. I know you're super active. I know you're super deep into this community and I just want to say thank you for uh spending so much time with me and um yeah, you gave me a lot to think about. Thank you so much for having me on. Really enjoyed this conversation. Phenomenal questions. Just we went really deep. Thank you so much. I I love this. Thank you.

Questions about this episode

What is discussed in "Sparks of AGI, Jagged Frontiers, AI forms "Internal Models" and the Dark Forest Hypothesis..."?

In order: 0:00 Dylan Curious, Sparks of AGI, Watson; 10:00 Agent Smith, Attorney General, Digital Twins, Replicator; 20:00 World government, Authoritarian regimes, Colossus, Memphis; 30:00 Dark forest hypothesis, Known universe, AlphaFold, Neural link; 40:00 150 characters, Attorney General, Bioengineering age, Black Mirror; 50:00 Democracy, Digital Twins, Actuarial tables, Authoritarian regimes; 1:00:00 Black Mirror, Ethics, Jeffrey Hinton, Wes Roth.

What is mentioned in this Wes and Dylan episode?

86 people, works and ideas are mentioned, including AGI (Artificial General Intelligence), ChatGPT, Step function (intelligence staircase), Artificial superintelligence, GPT-4, LLM. Each one on this page links to its own page with every episode that mentions it.

How long is "Sparks of AGI, Jagged Frontiers, AI forms "Internal Models" and the Dark Forest Hypothesis..."?

The episode runs 1 h 6 min; the transcript is about 13,211 words in 86 timestamped sections.

More from Wes and Dylan

• All analyzed episodes
• Books mentioned on Wes and Dylan
• People mentioned on Wes and Dylan
• Topics mentioned on Wes and Dylan