MIT Prof. Explains How AI Can (& Can't) Help w/Coding: Part 1

Who speaks in this episode?

• Armando Solar-Lezama (Main speaker)

What is discussed, and when

1. 0:00 – 3:00: Armando Solar-Lezama, MIT
2. 6:00 – 9:00: DreamCoder

Notable quotes from "MIT Prof. Explains How AI Can (& Can't) Help w/Coding: Part 1"

• I am a distinguished professor of computing here in the Schwarzman College of Computing at MIT, and I'm also the associate director of the Computer Science and Artificial Intelligence Lab, also known as CSAIL.

  about Armando Solar-Lezama · at 0:07

• And one of the things that we're trying to do with this DreamCoder project was to allow the system to discover its own building blocks, to identify common recurring patterns in…

  about DreamCoder · at 8:42

Who and what is mentioned in this episode?

People

• Armando Solar-Lezama — Person · Main speaker · 1 mention · first at 0:06
  He introduces himself and answers audience questions about how AI is changing programming, software engineering automation, and his research (e.g., DreamCoder).

Companies & orgs

• MIT — Company · Affiliation · 1 mention · first at 0:10
  MIT is the institution where the speaker, Armando Solar-Lezama, is a distinguished professor and associate director of CSAIL.
  I am a distinguished professor of computing here in the Schwarzman College of Computing at MIT, and I'm also the associate director of the Computer Science and Artificial Intelligence Lab, also known as CSAIL.
• CSAIL — Company · Research Lab · 4 mentions
  CSAIL is the Computer Science and Artificial Intelligence Lab at MIT where the speaker is the associate director.
  I am a distinguished professor of computing here in the Schwarzman College of Computing at MIT, and I'm also the associate director of the Computer Science and Artificial Intelligence Lab, also known as CSAIL.

Products & tech

• DreamCoder — Product · Research Project · 2 mentions · first at 8:05
  DreamCoder is a project by the speaker's research group that explores how AI can learn reusable abstractions in programming.

Full transcript with timestamps

0:00 (Logo whooshes) Hi everyone, very happy to be here. I am Armando Solar-Lezama. I am a distinguished professor of computing here in the Schwarzman College of Computing at MIT, and I'm also the associate director of the Computer Science and Artificial Intelligence Lab, also known as CSAIL. And I'm very excited to be here today to answer all of your questions about the future of AI and programming. And so let's start with the first question. How is AI reshaping the very idea of programming, what it means to write code?

0:38 And what remains uniquely human about that process as AI grows more capable? I think what's going to remain very human, regardless of whether you're writing research code or whether you are doing brand new development for a startup, is going to be this question of how do we actually tell the system, what is it that you're trying to do? What are you actually trying to accomplish? And this is where software developers bring a lot of their human knowledge and human expertise about, you know, what should this piece of software actually do?

1:19 And how do we break down what might be a very high level, very ambiguous statement of the goals that we're trying to accomplish with this piece of code down to a very precise description of step-by-step what needs to happen in this process. And now exactly how you do it and the nature of the notation and the tools that you do this are probably going to change quite dramatically in the next few years, just as they have changed quite dramatically, even in the time that, you know, I have been working with computers for example.

1:59 But I think what's going to remain very uniquely human is this creative process of figuring out how do we bridge from the needs to what we expect this piece of software to do in a very precise way. So question number two. Do you see programming shifting from writing code to simply expressing intent, where human specified goals and the AI handles the implementation? So I think one of the things to make clear is that already today for people doing software development, a lot of programming is about expressing intent.

2:41 We have a remarkable ability with modern programming languages to abstract away a lot of complexity so that the focus of the programming really is on expressing your intent. It is true that people who are starting to program often struggle with the syntax and the unnaturalness of the notation. But the fact of the matter is, once people get to the point where they're writing software for a living, the core challenge becomes expressing their intent at a level that is detailed enough and precise enough to actually build software.

3:18 However, there is no question that these new programming tools are going to allow us to be able to express things more concisely, to be able to lift the level of abstraction at which we describe things. And to get around some of those limitations of the current programming tools where sometimes you still need to write very large amounts of code to do what you want.

3:45 In your recent CSAIL study on autonomous software engineering, you identified key roadblocks. Which ones do you think will fall first? So making predictions about this technology is difficult because of how quickly the technology is developing. For example, one of the things that we've seen in recent years is this reasoning models that are able to think through challenging problems step by step, and in many cases, solve problems that they have never seen before in ways that are really exciting and surprising. One of the big challenges that we see in tools today is their inability though, to do this at scale, right?

4:32 And to do this at the scale of reasoning that is required when you're dealing with a large code base. However, we have a lot of tools coming from more traditional programming systems technology that can help us manage some of that scale and manage some of that complexity. And so I do believe that as we get better at incorporating some of that knowledge and some of those tools into the current AI based processes, we're probably going to be doing much, much better at dealing with larger and larger software systems. Now, this is just to one dimension in which current tools still have shortcomings, but I think it will actually make a really big difference in terms of using these tools in the context of harder, larger scale software engineering tasks.

5:32 Now, we have another question. How close are we to truly automated debugging, testing, and refactoring at the scale of real production systems? So all of these thing:; debugging, testing and refactoring come at many different flavors and many different scales. Debugging for example, you can have a bug that is, you know, just one line of code that is doing the wrong thing. And once you identify it and fix it, the software works. We have also seen bugs where you realize in analyzing the bug that there is a problem in the way you structured your computation and in the way you architected your solution that actually is going to make that bug really hard to fix.

6:26 And so the first kind of bug, I think we're actually already there, right? You can already use many of these tools to identify those kind of finger error type of bugs where all it takes is a small change in your code in order to fix it. And once you see it, it's very clear that there is a bug there. By contrast, the kind of bugs that arise from a faulty reasoning and faulty assumptions about your problem domain. I think we're probably still many years before we can have systems that can do that kind of debugging. And I think similar things have to do with things like testing and refactoring, right?

7:06 There are some refactorings that are very standardized and relatively simple and that can be implemented pretty mechanically even with tools that existed prior to LLMs. But the really challenging refactorings are those that really require rethinking the architecture of the code that require very non-standard changes that span the entire code base. And again, I think those are the kind of things that we're probably many, many years before we would trust a completely automated tool to do them. So I think to reframe these questions, I think it's not so much about the scale of the production systems, but about the complexity of the task that you're trying to perform.

7:57 So now we have a question here about the research that we actually do in my research group. The question says, in DreamCoder, you showed how AI can learn reusable abstractions. Could this lead to systems that invent domain specific languages on the fly? So for those of you who are not familiar with my research, let me give you a little bit of context. One of the challenges in software synthesis in generating programs is that you're often restricted to the building blocks and the abstractions that were already there present in the existing software, in the libraries that the system has already seen in order to produce the code.

8:42 And one of the things that we're trying to do with this DreamCoder project was to allow the system to discover its own building blocks, to identify common recurring patterns in the solutions that it generates, and abstract these into new components that can then make it easier to build new software in a similar way that humans build software, right? The first time around you might just write a whole bunch of code, but then you start thinking about how do I abstract this so that the next time I have to solve the same problem, I can build on the abstractions and I can build on these building blocks that I had before.

9:22 And so this is a really powerful capability to give our AI systems. And I do believe that in the future it will help us not just write code and write solutions for the problems we have at hand, but actually allow us to do more of what real software engineerings do, which is plan for the future, right? And plan for those abstractions that will not just help me solve the current problem at hand, but that will be useful to build the problems that I'm likely to encounter in the future. So the next question here is about natural language.

10:02 We talk to each other in natural language. We would really like to talk to our machines in natural language as well. And this is one of the things that this large language models really brings to the table. And so the question here is, what role do you see natural language playing? Could English or other spoken languages effectively become the new programming interface? So no. Being able to speak in natural language is great and you know, this is what we're familiar with from the time we were children. This is what we used to talk to our colleagues.

10:38 However, when it comes to building really big systems, natural language is very insufficient, right? If you've ever tried to read the terms and conditions of some of the software you use, you've probably run into some of the limitations of natural language when it comes to communicating things that are really detail oriented and complex, right? Any kind of legal document, you know, natural language very quickly starts feeling very unnatural once you start having to communicate things above a certain level of precision and above a certain scale.

11:15 And this is where programming languages really shine, right? As much as we like to complain about the fact that programming languages are too brittle and they're too clunky and it's hard to learn how to use them, the fact of the matter is, they are the best notation and the most sophisticated notation that we've been ever been able to come up with to describe really complex behaviors at scale. And so, no, I actually don't see natural language completely replacing traditional programming languages as the programming interface of choice.

11:53 It doesn't mean that there won't be a really important role for natural language in how we interact with our programming tools and how we talk about our goals with the software systems. But at the end of the day, it's really important to have that notation that can support the precision and the scale that we need for software development.

Questions about this episode

Who speaks in "MIT Prof. Explains How AI Can (& Can't) Help w/Coding: Part 1"?

Armando Solar-Lezama (Main speaker).

What is discussed in "MIT Prof. Explains How AI Can (& Can't) Help w/Coding: Part 1"?

In order: 0:00 Armando Solar-Lezama, MIT; 6:00 DreamCoder.

What is mentioned in this MIT CSAIL episode?

4 people, works and ideas are mentioned, including Armando Solar-Lezama, MIT, CSAIL, DreamCoder. Each one on this page links to its own page with every episode that mentions it.

How long is "MIT Prof. Explains How AI Can (& Can't) Help w/Coding: Part 1"?

The episode runs 12 min; the transcript is about 1,790 words in 18 timestamped sections.

More from MIT CSAIL

• All analyzed episodes