Transcript

CMU Introduction to Deep Learning 11785, Spring 2026: Lecture 1

From Carnegie Mellon University Deep Learning · Episode page · Watch on YouTube

· 1 h 23 min · 11,770 words · 110 timestamped sections

This is the full, timestamped transcript of "CMU Introduction to Deep Learning 11785, Spring 2026: Lecture 1", an episode of Carnegie Mellon University Deep Learning published January 12, 2026 (1 h 23 min, about 11,770 words). It mentions 12 people, works and ideas, including Neural Network, Artificial Intelligence, Perceptron, Connectionism, Hebbian Learning.

What is discussed, and when

  1. – 10:00: Neural Network, AutoLab, Kaggle
  2. – 20:00: Neural Network, Connectionism, Cognition, AutoLab
  3. – 30:00: Neural Network, Cognition, Google Translate, AlphGo
  4. – 40:00: Neural Network, Connectionism, Hebbian Learning
  5. – 50:00: Neural Network, Perceptron, Connectionism, Cognition
  6. – 1:00:00: Perceptron, Neural Network, Hebbian Learning
  7. – 1:10:00: Perceptron, Neural Network, Hebbian Learning
  8. – 1:20:00: Perceptron
  9. – 1:23:02: Neural Network, Connectionism

Notable quotes from "CMU Introduction to Deep Learning 11785, Spring 2026: Lecture 1"

  • So by now if you've been you should all have been signed on to Piaza.
    about Piazza · at
  • How it relates to cognition and the brain. We'll contrast it to conventional computer architectures. We will discuss early models of neurons and neural networks and their…
    about Neural Network · at
  • That's because in November 2016, Google switched over to using neural networks for translation.
    about Google Translate · at
  • In just a few hours they can learn to play go and become good enough to be the world champion. And this two is thanks to deep neural networks. By the way, someone just…
    about AlphGo · at
  • Investment. They even prove mathematical theorems. And so basically neural networks can do anything. If it's an AI task, the odds are you cannot beat neural network models. But what are these magical neural networks? From where we stand at.
    about Artificial Intelligence · at
  • Trillion connections which is what makes it so powerful. And so here's where we are. Connectionist machines are networks of processing elements.
    about Connectionism · at
  • Excites B, then its ability to excite B becomes stronger. So very pivily it's often stated as neurons that fire together wire together. Now here's how it actually works.
    about Hebbian Learning · at
  • Individual perceptrons are computational equivalents of neurons. The MLP is a layered composition of many perceptrons.
    about Perceptron · at

Who and what is mentioned in this episode?

Products & tech

  • Piazza — Product · Educational tool · 4 mentions
    Mentioned as a platform used for course logistics, communication, and polls.
  • Google Translate — Product · Case study · 2 mentions · first at
    Used as an example of how neural networks revolutionized machine translation.
  • AlphGo — Product · Case study · 1 mention · first at
    Used as an example of how neural networks achieved breakthroughs in game playing.
  • Siri — Product · Case study · 3 mentions
    Mentioned as an example of speech recognition technology before neural networks improved it.
    But after 60 years of research, the state of automatic speech recognition in 2015 was pretty bad. You simply couldn't use things like Siri, your typical speech recognition…
  • AutoLab — Product · Educational tool · 2 mentions · first at
    Referenced as a system for automated grading or lab work in the course.
    Canvas. In principle, the both systems should be getting the enrollment of the SIO and just enrolling you. If you find that you're not enrolled, please make a post and RTS will take care of it.
  • Kaggle — Product · Educational tool · 1 mention · first at
    Referenced as a platform for competing on state-of-the-art tasks in the course.
    The third part of the course is where you work on large scale problems. These are part twos of your homeworks where you'll be competing with each other on Kaggle on state-of-the-art tasks. So between these it's going to be a lot of.

Concepts & topics

  • Neural Network — Concept · Core topic · 26 mentions · first at
    Central theme of the episode, discussing the theory, construction, and application of neural networks in AI.
  • Artificial Intelligence — Concept · Core topic · 4 mentions
    Discussed as the overarching field where neural networks have become the main approach.
  • Perceptron — Concept · Key concept · 16 mentions · first at
    Introduced as the fundamental building block of neural networks, with historical and computational significance.
  • Connectionism — Concept · Key concept · 4 mentions · first at
    Presented as the theoretical foundation for neural networks, relating to how the brain stores information.
  • Hebbian Learning — Concept · Key concept · 3 mentions · first at
    Discussed as a foundational learning rule in neural networks, inspired by biological neurons.
  • Cognition — Concept · Key concept · 5 mentions · first at
    Discussed in the context of understanding human intelligence and how it relates to neural networks.
    So, today's lecture we'll begin with a brief history of neural networks. Under this, we will discuss the roots of connectionism and how it relates to cognition and the brain.…

Full transcript with timestamps

Good morning everyone. Can you hear me? Yes. Yeah, we can hear you. Good afternoon, Prof. We can hear you. Can

we wait a few seconds before we begin? All right, it's 8 in the morning. Good morning everybody and welcome to the spring 2026 edition of uh introduction to deep learning. So by now if you've been uh you should all have been signed on to Piaza. You should have seen a logistics post on Piaza. There's a lecture zero about logistics that has been posted on the course website. So you should have watched it. There are a number of recitation zeros which are background material for the course. You should have at least have uh looked through those if not if not watched some of them. You should have been to the course website.

The course website is deeplearning.cs.cmu.edu. If you've not been there, please visit it at once. Uh course objectives, logistics, quiz and homework policies, grading policies, all of these are both in the logistics lecture and on the course page. So please familiarize yourself with this information. Also again as I mentioned, you should be signed on to Piaza. You should all have been added to Canvas and AutoLab. Please verify that you have access to them. If you don't, post on Piaza and our TTAs will take care of it for you. You must also have seen the note on forming study groups. So, we recommend that students form study groups and work in teams because you work better in teams than you do by yourself. So, sign up for the study groups as soon as you can.

Third, our course philosophy is to leave no student behind. Now, it's a tough course. You're going to have to work very hard to get your A. But in our ideal world, every one of you would earn an A. And historically, about 70% of students in the course get some version of an A. But it's not not an easy grade. You work for it, but then we support you and we provide lots of resources for it. We have an army of TAs. We form you into study groups. Study groups are assigned TA mentors. We encourage collaboration. We hold uh over 50 office hours every week. I'm always available. You can email me and if I'm in Pittsburgh, you can just walk into my office. You're allowed to discuss and collaborate with your friends and classmates.

Also, the course is tough, but then you're also in the midst of going to be in the midst of a tough semester. So, some of you will find yourself under stress or unable to perform. If that happens, we have assigned some TAs specifically to deal with cases like these. So, reach out to your TA mentor, reach out to this team or to me and and we'll do our best to help you. Now, we really really encourage teamwork. So on Saturday the 17th we have a meet and greet plus a lab and a hackathon. There will be posts about it.

That's where you meet your colleagues, you make some friends and collaborators, you form study groups and if you already have a study group come as a group. You're going to learn how to get started with deep learning code and uh well there will be pizza. So I must mention that in this first week because next Monday is a holiday. So in this first week we'll have three lectures Monday, Wednesday and Friday so that we stay on top of the calendar and uh so the lab for this week will be on Saturday along with a hackathon.

Our course objectives is that by the end of the course you will all be able to understand some of the theory behind neural networks to be able to build your own neural network components and tools and to be able to work on large scale problems. So the first point understanding some of the theory behind neural networks you will hopefully know some of the what the why and the how. You'll understand some of the math. You'll provide some of the occasionally provide some history. And this will help you contextualize bullets two and three below. Namely that you will be able to build your own neural network components and tools and you'll be able to work on large scale problems.

So also understanding the theory will help you develop and extend your own ideas on the topic. This will help you in your research, will help you in your grad school applications and with job interviews. Also, by the end of the course, you should be able to build your own neural network components and tools. So, these are part ones of your homeworks and the homeworks also have various bonus problems. So, uh working on those will also give you a much firmer uh grounding and how neural networks work and how you can build build your tools yourself.

The uh third part of the course is where you work on large scale problems. These are part twos of your homeworks where you'll be competing with each other on Kaggle on state-of-the-art tasks. So between these it's going to be a lot of work but by the end of the semester you will probably find if you stay with us you'll probably find that you have learned more over this 3-month period than you did in any other 3-month period of your life. Some of you will have course projects. Those who are in 685 or 785 and course projects may relate to any of these three bullets.

My lecture style it's very I use lots and lots of visualization. I use many many slides. I animate stuff given a choice between deriving an equation symbolically and explaining it with 30 slides of pictures and animation. I usually prefer the visual uh the visual presentation with pictures and animation. If this is not the kind of lecture that you like then this might not be the class for you. We want you to attend all the lectures because you learn, but you kids will be overwhelmed with all that's going on in the semester. And so there is the temptation to maybe skip class when you can just get away with it. And we know that's not a great thing. So we have different ways of tracking your attendance. you get points for attendance and we will use in-class polls for class particip participation to ensure that you're there. There will be multiple polls posted at random times through the class for in-class lectures

where I'm holding the class in Pittsburgh. The polls will be posted on piaza. So the piaza post numbers for the polls will be called out in class and uh you should keep your piaza open. Then we have lectures on zoom like we do today. Then we have zoom polls and you must respond to all polls. We don't score you on correctness. We check if you've responded and we determine if you attended the lecture based on whether you show up whether and whether you responded to post. Now, some students are in really bad time zones and attending this lecture is going to be hard, especially if you're in the Silicon Valley where it's 5:30 5:00 a.m. right now. So, for these students and these students alone and for any other students who have received special permission, we allow you we allow you to watch MediaTek videos. All of the lectures are recorded. The recorded lectures go up on media media tech and on YouTube. Links to these

recordings will be posted on the course page and as it happens, MediaTek records who has watched the lecture and for how long and at what rate. So we can gather all of this information from there and we will use that to determine if you've attended the lectures. Again to reiterate, attendance is mandatory and we will track attendance to make sure you have attended. Now this is an interactive class. ask you like questions. My theory is that there's no such thing as a silly, wrong, or embarrassing question. So, if you have a question and you're feeling kind of squeamish to ask, don't feel squeamish because if you have that question, the odds are some hundred at least some hundred of the 300 students in class also have the same question. And so, if you ask them, you're helping not just yourself, but all of them. We'll try different tactics to encourage interaction. For example, calling out to students, please

participate. Now, for in classroom lectures, we try a few more tricks. We're going to assign each of you a Pokemon name. This is your secret ID for the class. During the lectures, I'll call out random Pokemon names and ask questions.

If it's your Pokemon, you must answer. Hi, professor. Yeah. Um, so I'm in the classroom. There's like 15 people here and uh I just set up the projection. So, they'll need a quick hello. And

that's okay. We're still just going over the uh logistics. So, uh maybe we can just go over this bit later. Right. And uh keep track of who's who's in class now. For Zoom classes, we're going to be tracking Zoom chat and responses to make sure that we answer all of your questions and uh we keep the interaction going. Then last but not the least, this is a dumb classroom, which means we expect you to keep all of your devices shut except the one that's watching this lecture on Zoom. You're not you're not supposed to be on Facebook, on anything else, doing anything other than paying attention to class. So in the classroom, no phones, no laptops, nothing is open except to answer polls or to view lecture slides or take notes. So if we find you using devices for any other purpose, if you're in the class, I may ask you to leave the classroom. On Zoom, I'm trusting you. This is an honor.

And finally, I'm handicapped. So, which means I'm twisted. I have all kinds of problems. I can't raise my arms. There are days on which I can't walk. I run out of breath. So, I have various ways of coping with this. In class, sometimes my clipon mic falls off. I ask my teachers to clip them on. I use all kinds of strange devices to point because they're light because I can't raise my arm. If you find these strange, please bear with me because I have no choice in the matter. So with that, I'll stop right now and I'll pause for any questions before we jump into the subject matter of the course. Do we have any questions?

Any questions? I was curious. Yeah. Uh I was curious just for clarification. So we can use our laptop to take notes, right?

If we're in person, laptop to take notes, but you know, I might just walk by you and make sure that that's all you're doing. One of the things that happens is that once I give one per person permission to open their laptop,

it's very hard to resist the temptation to go and sneak a peek at your email, to go and sneak a peek at your Facebook or whatever else. And it's not just you who's distracted. You're distracting everybody around you. And that becomes an epidemic. The whole thing can collapse. So, I try to keep that down.

Okay. Thank you. I have a question. Yeah. Yes. You said we should have been added to auto lab or something like that.

Yes. But for me, I didn't receive any email and even on canvas there is no announcement about that. So,

so there's no canvas. You should just be automatically added to autolab and canvas. In principle, the both systems should be getting the enrollment of uh the SIO and just enrolling you. If you find that you're not enrolled, please make a post and RTS will take care of it.

Okay. So, what about the study groups? You said that we should sign up for the study groups.

We have not already made a post with the about the study groups. We will be posting a form. you I encourage you guys to form study groups on your own but you must sign up and uh the form will allow you to mention the entire study group. Some of you will not actually have the friends or be able to form groups on your own. We'll make the groups for you. We try to make sure that everybody has a team to work with.

Okay. So the classes should should be should we be in the classroom for kegali students for example? Now for the Zoom classes, of course, you're going to be attending on Zoom,

but for the inerson classes, the classroom is has can only hold 140 people. So, everybody else has to be on Zoom, but we expect you to be on Zoom and keeping track and following the class and we can check.

Okay, thank you. Any other questions? Uh yeah, professor, I've not been enrolled into the piaza. So, am I expected to enroll myself or will I be getting an invite?

You should be able to enroll yourself, but if not, just email me or one of the TAs and we'll enroll.

Okay. Okay. All right. Any other questions? No. Okay. Yeah. Yeah. Love it again. So uh uh we have about 320 people enrolled in class as it happens there are only 230 20 people or 230 people on zoom

which shows you what happens that people stop attending they don't attend even at the very beginning of the semester. This can be bad because because the course won't wait for you, right? So, we will be checking attendance. Anyway, let's start with the lecture. So, today's lecture uh we'll begin with a brief history of neural networks. Under this, we will discuss the roots of connectionism and how it relates to cognition and the brain. We'll contrast it to conventional computer architectures. We will discuss early models of neurons and neural networks and their limitations. And then we're going to close by introducing modern neural networks and a discussion of what they can compute. At any point you may post questions on on the chat or speak up and ask me questions.

So why are we in this course? Well, artificial neural networks have now become the main approach to AI. They have been successfully applied to various pattern recognition prediction and analysis problems and in many of these they have established the state-of-the-art often beating benchmarks set using earlier machine machine learning techniques by very large margins and in some cases in many cases they have managed to solve problems that you simply could not solve using earlier methods. So how did where are we now? How did all of this happen?

We've been working on neural networks for a very long time since the 1970s actually much earlier as you'll see shortly. But the real breakthrough breakthroughs that brought us to today began happening in the 2010s with some kind of a flashover in 2016. So let me go through some of the breakthroughs in 2016. This was a turning point. Researchers had been working on automatic speech recognition technology since the 1950s. But after 60 years of research, the state of automatic speech recognition in 2015 was pretty bad. You simply couldn't use things like Siri, your typical speech recognition system, made too many mistakes to really be useful. Then in mid 2016, we had this breakthrough. For the first time, automatic speech recognition systems outperformed human beings on the switchboard task, which consists of people calling up each other on the phone and talking freely about various topics. The breakthrough was

made possible by artificial neural networks. And since then, the technology has progressed in leaps and bounds, and today SR systems generally outperform humans in speech recognition on most benchmarks. And speech recognition is now a staple user interface. This couldn't have happened without neural networks. Similarly, machine translation after 40 years of research, empty systems were horrible. In October 2016, if you put an English sentence into Google Translate, which was the best translation system at the time, translated it to Spanish and then translated the Spanish back to English again, what went in and what came out wouldn't look anything like each other.

We used to we used to use this as a party trick. It just didn't work. Then in November 2016, something amazing happened. Google Translate suddenly became really good. so good that even the professional translators began using it for the first pass of translation. That's because in November 2016, Google switched over to using neural networks for translation. The technology has continued to improve and these days uh automatic machine translation systems are better than all but the most expert human translators for most languages. And this was made possible using neural networks or uh the basic raw intelligence. For the longest time, we assumed that the ultimate challenge in AI was playing mental games like chess and go. And to the 1990s, automatics chess games were horrible. They didn't really do do very well. I don't know how many of you know this but the first chess game to beat a grandmaster was built right here in CNU.

This was a deep thought by Feng Shingu and that was his PhD cases. Then from there he went to IBM where in 1995 he built the first uh automatic chess game that beat a world champion and after that no human has really managed to beat the best computer chess games. So yeah, we conquered chess, but then we still had go and go was was generally considered as something that computers could never really get good at. Go is vastly more complex than chess. A chess game has 10 raised to 120 possible states, but Go has 10 to 180 possible states. So it was assumed a computer could never beat a human. Then in 2016, Alph Go shocked the world. could beat the world champion.

Today, uh you have programs that don't even need to know the rules and can begin learning from scratch on their own and in just a few hours they can learn to play go and become good enough to be the world champion. And this two is thanks to deep neural networks. By the way, someone just uh requested permission to record I think using Otter. I denied it because when I do that, when I allow per when I grant permission, what happens is that the actual recording of the class that I've triggered on Zoom that switches off sometimes and then we lose the class recording. So, you might have have to uh bear with me on that.

So, here's another example. Image captioning. This is also an old result from 2016, but here for the first time, computers could automatically caption an image like a human. So, all of these captions were automatically generated by a neural network. Man in black shirt playing a guitar. Construction worker in orange safety vest working on a road. Two young girls are playing with a Lego toy. Boy doing backflip on the wakeboard. Now, these are shockingly accurate captions. You might think it's kind of normal, but it's not. And before 2016, something of this kind was considered basically impossible.

All of these were produced by a network that only analyzed the image itself and no other input. And this kind of intelligence only became possible in 2016 thanks to neural networks. Of course, modern models are far more accurate. So, here uh this one, this is a neural network model based AI system that analyzed this picture taken by my ex teaching assistant Current. He just presented this picture. He took up his room and asked Chad GPT about it. And here are all the things it manages to tell him. It tells him the ceiling in the photo appears to have several noticeable water stains which could indicate potential water damage. It says there are several posters of prints on the wall featuring various images likely of scenes from movies. It's true. The room contains a few pieces of furniture including a blue chair, a black wardrobe, a storage unit, and a desk with two monitors.

The ceiling has a light fixture and an air vent suggesting this is a standard room with typical building amenities. The lower part of the photo photo shows a bed with a person lying on it with their foot visible in the foreground. The bed has a comforter and some clothing or linens on it. Uh the overall condition the room appears somewhat cluttered with personal belongings spread out. Amazing. And this was just done using a neural network. This was a couple of years ago. These days they're even smarter. So as you can see neural networks are getting really, really, really smart. And they are the solution to so many problems from art to astronomy to health, chemistry, physics, biology, business management, investment. They even prove mathematical theorems. And so basically neural networks can do anything. If it's an AI task, the odds are you cannot beat neural network models. But what are these magical

neural networks? From where we stand at this point in the course, they are just these magical boxes where things go in and other useful things come out. For example, a voice recording goes in, its transcription comes out. An image goes in and its caption comes out. Maybe a game state goes in and the next move comes out. But what's in the box? To understand this, let's begin here. All of these tasks that we just looked at, recognizing speech, captioning images, playing complicated games, all of these are fundamentally human activities powered by human intelligence. So perhaps the place to begin is with the seat of all human intelligence, the human brain or even earlier with our amazing ability for cognition, our ability to think, to analyze, to create.

So maybe that's where we want to begin. Understand how that happens. So here are all the magical cognitive capacities we've got. We can learn. We can solve complex problems. We can recognize patterns. We can create. We can just sit in one spot and coach, think and reflect and imagine and dream without any external stimulus. So these are the abilities that make us who we are. So if we want to build a machine with humanlike capabilities, these are what we must emulate. But then how exactly do we function? How does the brain perform all of these activities?

The problem is that the brain is an immensely complex organ. It has to be to power our stupendous intelligence. So anything less complex cannot generate the kind of intelligence we have. As a result, just quoting Marvin Minsky, if the brain was simple enough to be understood, we would be too simple to understand it. And basically the complexity itself comes in the way that generates intelligence comes in the way of our ability to understand it. Nonetheless, we're going to try. Now, people have been curious about human cognition for thousands of years.

And in fact the earliest models for cognition go back to at least the ancient Greeks. The earliest known model for human cognition is called associationism. And we know that theory goes back at least to plateau in 400 BC. So this picture here is the famous school of Athens that you can find in the 16 in the cyine chapel and this painting is by Rafael. It shows all the scientific grades from Greek times until the 15th century. And here in the center you have Plato and Ari Aristotle. So Plate was the first to actually record some kind of a theory for how human brains work and this was associationism.

Association in ism in fact remained the primary model for human cognition for from Plateau's time right until the early 20th century with scientists like David Hume and even uh as recently as Ivan Pavlo developing theories around it. Associationism posits that the process of cognition is fundamentally one of learning associations. So what are associations? These are literally percepts that we somehow associated with one another to make inferences. For example, in our experience, we find that lightning is generally followed by thunder. So we make associations between lightning and thunder and not just an association of the events but also a temporal association. So because usually lightning is followed by thunder. When you see a bolt of lightning, you expect to hear hear thunder shortly.

Conversely, if you hear thunder, you assume lightning has just struck somewhere nearby. These are associations. And you're of course familiar with the most famous association experiment of all, Pavlo's dogs. Now, where he showed that after enough training, every time he rang a bell, the dog drooled. There are many more slides on associations in the slide deck. Please look at them. I'm not presenting them yet, but they'll appear in the quiz. So we have this theory. We think through the process of association and it's a really good theory and we still use it in machine learning implicitly. But then the question is how and where are these associations stored in our brain?

There were several sometimes strange theories on how the brain actually stores associations. But things began to get more real once we learned about the structure of the brain itself. And this happened with the invention of good microscopes using which by the mid 1800s we had figured out that the brain is essentially a large mass of a special kind of cell which we later call the neurons. And these neurons are all connected to to one another. Each neuron has many neurons connecting into it and each neuron in turn connects out to many neurons. So the brain is essentially a network of neurons and somehow this network of neurons captures all of our all of our capacity to be intelligent.

The first person to realize how things worked in the brain was Alex Bane back in the 1870s. Like all the scientists of the day back then, Bane was a bit of everything. He was a philosopher. He was a psychologist. He was a mathematician. He was a logician. He was a linguist. Last but not the least, he was a professor, which was much more difficult to be back then than today. Now in his 1873 book mind and body postulated that the brain stores information through its connections he he basically postulated that all of the information that's stored in the brain regarding associations between percepts or more generally between stimulus and response is captured through the structure of the connections between neurons. So this was in fact the first connectionist model and he even came up with various computational models for for this theory. The key idea here is one of his models. The key idea here was

that the same connections if appropriately arranged could result in different outputs for different inputs. This may seem very obvious to us today but in Bane's time 1873 it was considered something really weird. So here's one of the circuits here. Again, we assume that all the inputs and all outputs are binary either zero or one. And a neuron fires if all of its inputs fire. So here if A and B fire, X fires. If B and C fire, Y fires. If A and C fire, Z fires. So although you have a single circuit for different input patterns, you get different firing patterns for the output.

And Bane even even included models where a circuit's output could depend on the intensity of the input. For instance, over here, if the input is low intensity, then X gets only two copies of it and may not fire. But Y gets three copies of it if the input persists long enough and Y will fire. On the other hand, if the input is high intensity, both X and Y will fire. So these this models like these are in fact very modern neural network models. Bane came up with them in 1873. over over 150 years ago. So these models have been out around for that long. And Bane not only showed the circuit structures, he also proposed a learning mechanism whereby the brain or any computational model of the brain could learn these association, these connections. And here's Bane's proposed mechanism for how neural and neuronal connections could be learned.

In his own words, when two impressions concur or closely succeeded one another, the nerve currents find some bridge or place of continuity, better or worse, according to the abundance abundance of nerve matter available for the transition. So this is literally Hebian learning which you'll hear about in a few minutes. Heb came up with a model in the 1940s, but Bane theorized about them almost 75 years before Donald. That's how ahead of his time he was. And being ahead of your times can often be a bit of a curse. And so sadly, Bane's peers in the 19th century thought he was speaking nonsense. And eventually even he began doubting himself. To quote Bernard Bernard Russell, the fend of fundamental cause of the trouble is that in the modern world, the stupid are cockure while the intelligent are full of doubt. And so Bane doubted himself.

In 1873, Bane postulated that there must be 1 million neurons and 5 billion connections to capture 200,000 percepts. This seemed like a very large number. But then by 1883, he realized that this had not taken into account the number of partially formed associations. And to account for these, he'd need even more neurons and connections. So by 1903, he decided that our brain was simply not large enough, that his theory was wrong. It would need too many neurons and too many connections for his model to work. And so he apologized and he recanted all of his ideas which is which is a tragedy because he was correct in 1873 and he was wrong at the end. Our brain is very large. It has 80 billion neurons and about 100 trillion connections. That's comfortably more than any capacity Bane assumed was required. And his model is in fact more or less how we think our brain works today.

And so connectionism lives on. We know by now that our brain is a connectionist machine as predicted by Bane in 1873 and David Fer in 1876. Neurons connect to other neurons and the processing and the information capacity of the brain is a function of these connections. And so modern artificial neural networks are connectionist machines which emulate the structure. So this is what a connectionist machine is. It's a network of simple processing elements where all world knowledge about the task the the machine performs is stored in the connection between the elements. The elements themselves are very simple and modern neural networks are thus connectionist machines consisting of many little processing units that connect to one another. Now uh does does anybody remember what is the type of architecture that uh your standard processor has?

Does anybody remember the name of the architecture? I'm sure you have some computer science students over here who remember this. Anybody? It's a v oneman architecture.

So what is a oneman architecture? Anyone? Okay, someone answered but yeah, so in a in the it's a variant of the so-called one architecture in which you have a processing unit and you have a separate memory where programs and data are stored. There are other variants like you have the Harvard architecture in which you have separate memory for programs and data and in the original one I mean architecture you had this common memory for both programs and data. Uh modern machines use a hybrid approach. The processor is Harvard architecture but the main memory is the one I'm in now Princeton architecture. Now those differences are not important. What is important for us is the form of the architecture itself. your processor is separated from the memory which holds the programs and the data. So if you want to run a new program, all you have to do is to upload a new program into the memory and you're set to go. Which

is why your typical computer or even your phone is able to do so many different things. In order to make it perform a new task, you just have to load the corresponding program into the memory and you're set to go. That's not the case with connectionist machines. for connectionist machines because the program is in the connections. The computer itself is the program. The program and its memory are all encoded in the connections between the units. To change the program, you have to change the machine itself. Which is why you don't you very rarely build hardware neural networks with everything built in because if you wanted to change the program, you'd have to build something completely new. Instead, we typically emulate the connectionist machine on regular Harvard and Princeton architecture machines.

And so to recap, neural network based AI has taken over most AI tasks. Neural networks originally began as computational models of the brain or more more generally as models of cognition. The earliest model of cognition was associationism. The more recent model is connectionist. Neurons connect to neurons and the workings of the brain are encoded in these connections and current neural network models are connectionist machines. So we now have our first poll. Uh let's see. Can you folks see the poll?

Okay, 10 seconds, guys. And listen, the slides must be on the course website. Okay, let me end the poll. And does anybody want to answer the first question? What's the answer?

Alexander, that's B, right? And for the second question, how many connections did we say exist between neurons of the brain?

80 billion. 100 trillion. Right? The correct answer here is 100 trillion. So uh uh Alex Bane was the first person who proposed connectionism and the human brain has about 100 trillion connections which is what makes it so powerful. And so here's where we are. Connectionist machines are networks of processing elements. All the information the machine uses to perform its computation is in the connections. But what are these individual elements themselves? To understand that, let's go back to the brain. The individual units in the brain are neurons. A neuron looks roughly like this. It has a head called the soma within which the nucleus resides. The soma has several dendrites through which it receives signals from other neurons.

The neurons the a neuron generates signals that are spiked trains and the rate of the spike train depends on the accumulated signal from the other neurons. At a high level, we can think of it in this simple way. If the total input signal from all the other neurons exceeds a threshold, this neuron fires. That output signal then then travels down this long leg called the axon from which it's transmitted to other neurons. Now, here's an interesting factoid. This axon is very long. And so, it has to be protected for the neuron to work. and it's covered for protection. It's covered with a sheet sheet of fat called the myin sheath. This fat is crucial for the functioning of the brain. In fact, scientists who studied this sort of thing have found that the key distinction between Einstein's brain and normal brains was not that he had more neurons in his head, that he had more gal cells, the cells that produce the

smile, the sheath that protects the axon, than normal people. So it appears that the more fat you have in your head, the smarter you are. And in fact, being called a fathead is actually a compliment. Now, here's another factoid that may only interest me. It should interest you also. Neurons don't undergo cell division. Why? Because all the information in the brain lies in the connection between neurons. If the neuron divided, these connections would get destroyed and you'd lose that information. So a neuron once it's in place cannot divide.

New neurons instead are created from neural stem cells and neurogenesis which is the formation of new neurons begins to slow down almost as soon as you're born and pretty soon goes down to something close to zero. In other words, you're born with almost all the brain you'll ever have. Anyway, for our computational connectionist models too, our compute units are going to emulate the nature of this biological neuron. And the first computational model for the neuron was proposed by these two guys, Melo and Pitts. Warren Melo was a neuro physiologist in the University of Chicago. Walter Pittz was a homeless guy who just turned up and knocked on his door. So I'll let you figure out which of these is Walter Pitts and which of these is Melo. Who do you think is Pitts over here? The guy to the left or the right? Anyone?

Nobody want to take a guess? Right. So this hobo looking the hobo looking guy over here is actually Warren Mccelo and Walter Pittz was a very young guy. He was only 20 when he knocked on melo door. He was a runaway from run away from home and he stayed with melo for a few months and they came up with this amazing model called the mechlo and pits model of the neuron. It was published in 1943 in this paper a logical calculus of ideas imminent in nervous activity in the bulletin of mathematical biohysics physics. So this picture on top is a pictorial visualization of the model.

Now their model was a breakthrough. how it was the first computational model for a biological process, namely the neuron. And so here is where began the entire science of modeling biological elements through mathematical models. In their model, they modeled the neurons in the brain as performing propositional logic, basically boolean logic. And so by modeling it as performing propositional logic, they suddenly give us attractable mathematical formalism to represent the brain's operations. This is a little more detailed picture.

So the actual neuron itself is shown by the triangle. The line going out represents the outgoing signal. The colored lines represent incoming signals from other neurons which connect to this neuron through these synaptic connections. There are two types of sap synapses. The excitatory sinapsis shown in green and the inhibitory sinapse shown in red. The excitatory sinapse transmits signals weighted signals from other neurons to this neuron. So the number of connections over here represents the weight. Now this causes the neuron to trigger and fire. But if there's any signal coming up the inhibitory connection then this absolutely prevents the neuron from firing regardless of what is coming in through the excitatory signal. If the inhibitory input is turned on the output of the neuron will always be turned on.

But now by appropriate choice of excitatory and inhibitory synapses the mecho and pitch neuron can be made to emulate any boolean gate. So these are shown here. In each case, the neuron shown by this triangle here fires if the total input is at least two. So in the top left, if neuron one fires, then one time unit later, neuron 2 gets two inputs and so it fires. Over here, this neuron responds to one and two. If this were a boolean function, what is this boolean function? Can anybody tell me what is this boolean function?

This one here. It's an and. Both one and two have to fire. That's when three receives two inputs. What about this guy? Yeah. What is this?

Why? That's an R because if either one or two fire, three receives two inputs and so the neuron fires. And this is a little more complex. one and not two because if two fires two is inhibitory, right? And so you can see that you can construct pretty much any boolean gate using the perceptron and so you can construct any boolean circuit using these neurons but you can do a whole lot more. Meal and pits also showed that you can produce complex responses. For example, there's a very well-known phenomenon in human perception. If you touch something cold very briefly, just touch it and take your finger off, it doesn't feel cold, it feels hot. But if you keep your finger on it for a little longer, then you will feel the cold. This is well known. And they showed that you can actually model this using the structure.

Here I have two inputs, neurons one and two. And these are a heat receptor and a cold receptor. This blue unit is just a delay unit. It basically reflects whatever the cold receptor got but one time unit later. Now unit four is the cold response. Unit three is the heat response. At the initial time all outputs are zero. Now if the cold receptor fires at say time t= 1 then one time instant later the cold response unit cold sensation unit gets one signal from two. But then if you're if you touch the coal receptor very briefly then one time second later this guy this guy gets a signal but at that time this has not fired. So you're not going to feel the cold but if you keep touching the cold object for a long time at time one this sends something down. At time two also this sends something down to the cold center cold cold sensor. Now this guy reflects because it's a delayed version of two. It reflects what it's what the

cold receptor saw one time instant ago. And so if the coal receptor sees perceives coal for two continuous time instance then the cold sensation neuron gets two inputs and it fires. On the other hand if you just touch the coal sensor for only one time unit then this at the first you know this guy gets one input one time instant later. It gets one input two time instance later. Neither is enough for this to fire. But then what happens here? If you just touch it for one time instant, one time second instant later, this fires. Another time instant later, this fires and then eventually this fire. So you feel heat. But if you keep touching the cold object for an extended period of time, then at the second time instant, this inhibitory signal comes in and prevents this guy from firing. And so the heat sensor doesn't go off. And so you have this model which captures this human phenomena that a brief touch of cold

makes you feel heat but if you keep if you're persistent uh uh input of the cold sense to the cold scepter will actually make you feel cold. So clearly you can see that this meal model is quite powerful but then it had a lot they faced a lot of criticism. First, they claimed that the network could compute a small class of functions. This is true. They claimed that if you provide the network with tapes, they would become Turing complete. That they could solve any problem that a Turing machine could solve. Also true if you don't consider the size of the machine.

But they didn't actually prove several of their results. These were just claims. And most importantly, they didn't provide any learning mechanism whereby the network could learn its connection weights from experience from data. For that we had for to wait for another guy. This guy Donald Heb who I mentioned a little while ago. Now Heb was another interesting character. He was a Canadian. He initially trained to be a novelist. Then he decided what he really wanted to do was to be a farmer. Then he threw that away and decided to become a hobo. Then he figured he'd be a school teacher. And then he said, you know, screw it. No, no, this is working for me. And so he went and joined Harvard and became he became a psychologist and got a PhD PhD from Harvard and then went on to work in a variety of places. So in 1949 he published a book called organization of behavior where he outlined a mechanism whereby neuron and neuronal

cells in the brain could learn their connection with experience from data. So here's how we describe the mechanism. When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency as one of the cells firing B is increased. Sounds complex but basically what he's saying is that if A connects to B and A repeatedly excites B, then its ability to excite B becomes stronger. So very pivily it's often stated as neurons that fire together wire together. Now here's how it actually works.

Let's look at one connection between two neurons. Let's call them X and Y. Now neuron X connects to neuron Y at a sinapse. At the sinapse, the axonal connection from neuron X has a knob called the synaptic knob. And the two neurons don't actually touch one another. What happens is that uh when a signal comes down this neuron, some chemicals called neurotransmitters are are emitted from this head which go and trigger the second neuron which gets its input. Now, every time a signal from X successfully triggers the the the second neuron Y, then this head gets a teeny weeny bit larger, which means that the next time around a signal comes down from here, it becomes easier for this head to trigger B. So, anytime one neuron excites another successfully, it makes it easier for the neuron to excite the second neuron the next time. Now, this can be formulated in that Let wxy be the weight of the connection

between neuron x and neuron y. In this model, neurons are binary and the output is either zero or one representing whether the neuron is firing or not. So the heavy learning rule says that at any time both x and y take the value one meaning neuron x is successfully triggered neuron y wxy the weight of the connection increases. So you can write it in math as wxy = wxy + beta * xy. See that this increment note that this increment only happens if both an x and y take the value one. So if a neuron successfully triggers another neuron the weight of the connection increases. This simple formula called the heavian learning rule is actually the basis of many learning rules in machine learning. But then there's something problematic with this.

Can anybody identify what the problem is is over here? Anyone? What's the problem with it with with this with this formula? Anybody?

Looks like linear infinite growth. It's infinite growth. There's no negative learning. It keeps increasing and increasing and increasing. So, it's fundamentally unstable, right? The weights can only increase and never decrease. So over time eventually all the weights will become large and the entire model is going to become useless. And people came up with various proposals like uh Sanger's rule and whatnot to try to make uh to try to sort of correct this and make this work in a more uh reasonable way. But the best and the most lasting update to the Hebian rule came came with the introduction of a new model. But before that I just released my next poll.

And then we'll go back to the improvement to He's Okay, 10 seconds, guys. All right, let me end the poll. So pretty much all of you got the answer right. But does anybody want to tell me what the answer is over here?

It's unstable because there's no reduction in weights. Exactly. And so we need an improvement to this model. And that improvement was proposed by this guy Frank Rosenblot. He was a psychologist and logician at Yale. And uh he actually is the father of modern neural networks. He died at the age of 40 unfortunately in a boating accident where he went boating to celebrate his 40th birthday. Anyway, uh he in 1958 he proposed a model called the perceptron which at that time he was sure would solve all AI problems. Here's the full picture of Rosenbl's perceptron. It's actually a very complex structure composed of many units arranged in many layers with epherent and apherent and epherent connections and whatnot. But then here's a simpler visualization which captures its essence. So Rosenlot's original proposal related to perception in the eye and uh so the details of it are in the slides but in

the simplified the simplified visualization uh is something like this. These over here are the sensory units. These would be the cells in your retina. Then you have something called an association layer with many associative units which consists of many neurons each of which connects to the sensory units. And then finally you have a layer of response units and these response units connect to all of the associated units. And this entire structure is what Rosenblot called the perceptron. In modern parliament when we speak you know these days when I speak of a perceptron I'm only speaking of uh say this one guy over here each of these individual units is what we would call a perceptron but when Rosenl first introduced this he called this whole thing was called a perceptron anyway let's see what each of these units looks like just this one guy how does it uh uh what does it look like So here that unit looks something like

this. Each neuron receives a number of inputs x1 through xn. Corresponding to each input is a weight wn. It computes a weighted sum of all of these inputs. And the weights can be either positive or negative. If the weighted sum exceeds some threshold t, the neuron outputs a one otherwise it outputs a zero. So in equations we can write it like so. If the weighted sum minus the threshold is positive then the neuron outputs a one otherwise it outputs a zero. So electrical engineers will recognize this unit as a threat as threshold logic performing threshold logic.

Rosenblad also provided a beautiful learning algorithm for how these units could learn their weights. And so here let w be the weight with which any input x uh is connected to a perceptron. After each new input the weight is adjusted to account for the error between the actual output of the perceptron and the desired output of the perceptron. So in math we can write it as w = w + a * d - y d of x - y of x * x where where y of x is the actual output of the perceptron and d of x is the desired output of the perceptron. Does anybody see the similarity between this thing and and the the heavian rule?

Do you see the similarity between this and the heavy rule? How does it differ from the heavian rule? anyone.

It has a loss function so to speak. It has a loss function gives negative feedback. Here's the thing in rule. This guy was just y which which increasing and increasing and increasing. Here you introduce the notion of a desired output and if the output is already equal to the desired output the weight does not change. It changes not in proportion to the desired output but in proportion to the error from the desired output. That was the critical difference between this and the original Hebian rule. And so this Rosenlot not only came up with this idea, it's a beautiful learning rule and he also showed that if you have linearly separable data, meaning the data for which uh if you have data for which the desired Y is one and other data for which the desired Y is zero and these data can be separated by a hyper plane.

Then this learning rule will is guaranteed to converge to give you a set of weights that will separate the two sets of logs. We'll talk about more about this later. Right now, Rosenblot originally assumed correctly that with the appropriate choice choice of weights and a large enough number of units, this full model could represent any boolean function and perform any logic and the popular media of the time bought into the hype. So, here are some snippets from a couple of newspapers at that time back in the 50s. The embryo of an electronic computer that the Navy expects will be able to walk talk see write reproduce itself, and be conscious of its existence. 1958.

Frankenstein monster designed by the Navy that paints. 1958. Now, I mean, we're getting there, but in 1958, we were 70 years away. And that wasn't just it. The problem was Rosenblat made claims about this model. But by and by people began assuming that the claim also held true for the individual units in the model which are these guys right and that these guys themselves were what we'd call universal boolean machines that could compute any boolean function. No, this is not true. And as a result of this exaggeration, as a result of this exaggeration, research funding in neural networks dried up for a decade and uh research almost came to a standstill. But for now, let's look at the capabilities of this individual unit, what is now called a perceptron.

In fact, even the simple unit is very powerful. It's relatively easy to see how this perceptron can mimic any boolean function. Consider these examples here in the circle. uh I have a threshold and the numbers above the arrows represent the weight. So if the total weighted input matches the threshold the neuron will fire. What is this gate over here? Anybody? What is this function here? Top land.

And what about this one here? or or and this guy not not that's a not if x is one the input becomes minus one and it fires but if x is zero you know if x is one and it doesn't fire but if the x is zero it fires right so this is a n so you can see you can model any boolean gate and and but then the limitation of the individual units becomes apparent when we consider the exort there's no setting of the weights for which the unit can compute X or Y and we'll see why later right so in any case this means the perceptron is not a universal machine boolean machine how do you resolve this to resolve this we go back to the brain the brain is not just a single neuron it's a large network of neurons so for our computational model too the individual neurons perceptrons are weak they are not enough but you can greatly extend them to do what you want once you begin networking many of them.

And so if you just connect up three of these perceptrons like this, we can compute an XR. Here we have connect connected three perceptrons in a layered manner. This top one connect computes X or Y. This computes not X or not Y. This ends the two. And so now you have the X order, right? Some terminology over here. These two neurons, we call them the hidden layer. Not because they're really hidden, but we because we are not really interested in the individual outputs. We are interested in the output of this final neuron which we call the output neuron or the neurons in this layer called the output layer. Anyway, so once you realize that you can begin connecting up uh perceptrons into a network to compute more complex boolean functions, then you realize you can compute arbitrarily complex boolean functions simply by connecting up perceptrons in the right way. And so in cognitive terms, we can compute

arbitrary boolean functions of sensory input. So for example, this network here computes this crazy function of four inputs. It doesn't matter what kind of boolean function you give me. I can always construct a network that computes of perceptrons that computes that boolean function. And so a multi-layer perceptron, this is what we will call a multi-layer perceptron because you've connected up the perceptrons in layers. They're actually a universal boolean function. So story so far, neural networks began as computational models for the brain.

They're connectionist machines. They comprise networks of neural units. The mecho and pits model had neurons as boolean threshold units which models the brain as performing propositional logic. They gave learning rule. Heb came up with a learning rule. Neurons that fire together wire together or it was unstable. Rosenlot's perceptron which is a variant of the mechoic neuron had a provably convergent learning rule but the individual elements in it are limited in their capacity but multi-layer perceptrons can model arbitrarily complex boolean functions they are what are called universal boolean functions but then our brain is not boolean everything we've discussed so far is about boolean inputs and outputs our brain operates on real valued inputs and often makes real valued or nonv boolean inferences. So how does the perceptron square this?

First let's consider real valued inputs. Again the weights two are real valued. The perceptron fires if the weighted sum of inputs exceeds the threshold t by fires it may say the output is one otherwise the output is zero. So this is your perceptron. I can redraw it slightly like so. Instead of having a threshold t over here, I can say the perceptron first computes an aphine function of the inputs which is a weighted sum of the inputs plus a bias term and then it passes this aphine value through an activation threshold function which fires if this value coming in is positive and does not fire outputs a zero otherwise. So can you see how these two are the same model with just a small change in terminology?

Can can you can you guys see that these two are the same thing? Yes or no? Anyone guys it's awfully quiet. There's just one person who sees that they are the same thing. So what is the confusion here? Right. So instead of saying I'm going to be applying saying the weighted sum might should be greater than the threshold I can say I computed a weighted sum plus a bias where the b here is minus t right and then I put it through a threshold activation the the bias term basically represents this threshold Jeffrey that answer you this minus t and this plus b are the same that answer there's a question on in the chat.

Yeah. Yeah. That's the one. What does the bias term represent? Right? That's the one I'm talking about. So this bias this, look at this here. I'm saying summation wxus t that's summation wx plus z. So this is the threshold. Anyways, so this is an aphine function of the input. Does anybody know the difference between aphine and linear? What's that? What's the difference between Afine and Lia? Anyone?

Um, what's the difference? Aine Aine is not reversible. Anybody else? Non- negative. Okay. So suppose I have a bunch of inputs and I say some and I call the summation wixi. If I equate this to zero, this represents the equation of a hyper plane that goes through origin. If all the inputs are zero, then this becomes zero. This is a linear function. On the other hand, if I have summation wi xi + b, if I equate this to zero, this represents the equation of a hyper plane. It's still a hyper plane but it no longer goes through necessarily goes through origin. So this is an aphine function. You see the difference between the two linear versus aphine.

Yes or no? Right? Okay. And so you first compute an aine function of the input and then put it through a threshold activation. And once you realize you can put it through a threshold activation then you can which generates outputs of either zero or one. You can have other activations functions and uh this uh allows the output to be real valued rather than binary. But in any case for the purpose of building our intuition and interpretability I'll continue to assume the threshold activation for now. So here's the real value perceptron again.

It operates on a real valued space of inputs. If the weighted sum of the inputs exceeds a threshold, the output is one. Otherwise, it's a zero. And the boundary between the two is where the weighted sum exactly equals a threshold. And so that is going to be a hyper plane. So the overall function is a heavy side function of this kind. There's a boundary. On one side of the boundary, the output is zero. On the other side, the output is one. So is everybody seeing this how this is true? Yes or no? This make sense to you guys? Okay. So once you realize this then you can see model boolean functions. When the input is boolean then if you have two inputs the two inputs can take values 0 0 1 1 or 1 0. I can always design my weights such that this boundary lies here. If this boundary lies here, what is the function I've captured over here? What is the boolean function

over here? Anyone? Or or? That's an or right. What is so if I put the boundary here, what is the boolean function?

And that's an and if I put the boundary over here, what's the boolean function? not

that's a not right and so now can you tell me why you cannot do an exor I'll let you think about it we'll look at this in the next class right and so but now once you see how a single perceptron behaves you can immediately see how to network them to construct arbitrarily complex decision boundaries for example if I want a network that looks at two-dimensional inputs but produces a one Only if the input is in this pentagon and zero outside. Here's what I would do. I'd have one perceptron that captures this boundary. It outputs a one in the yellow region and a zero in the zero. Zero outside. Another perceptron for this boundary. A third perceptron for this boundary. A fourth perceptron for this boundary. And a fifth perceptron for this boundary. And the pentagon is the only region in which all five of them have the output value one. So I can have a final perceptron which just compares the weighted sum to

five. And this network is now going to output a one if the input is inside the pentagon and zero zero outside. So you managed to capture this pentagon. But once you know how you can how to construct uh a network or a pentagonal decision region, you can make networks for more complex regions like this double pentagon. We want the network to output a one if the input is inside either of these pentagons and zero zero outside. So you can have one subnet which outputs a one inside this lower pentagon, another one which outputs a one inside this upper pentagon and a final neuron ors the two. And so now this o overall network is going to output a one if the input is in either of these two pentagons and zero outside.

Is this making sense to you guys? This picture? Yes. No. So okay. So once you know how to do this, you can do more crazy decision boundaries like this, right? Can be arbitrarily complex. And so uh you know we know now we know how to build multi-layer perceptrons for complex classification tasks. Your inputs are now high dimensional. For example, if you if you were building a network to recognize digit two in the eminest digits, your inputs are 784dimensional, the region of the input space that corresponds to the digit two will have some shape. Maybe like this peanut here.

And so all you need to do is to build an NLP that fires that outputs a one if the input is in this region and zero outside. And so you have a classifier. And so here's the story so far. MLPS are connectionless computational models. Individual perceptrons are computational equivalents of neurons. The MLP is a layered composition of many perceptrons. They can model any boolean function because individual perceptrons act as act as boolean gates and networks are of perceptrons are boolean functions. So MLPS are universal boolean functions.

MLPS can also model any decision boundary because individual perceptrons capture linear boundaries and complex boundaries can be composed from these linear boundaries. So MLPS can represent arbitrary decision boundaries which means they can be used to classify data. So MLPS are universal classifiers. Uh here is your third poll. I'll give you only 30 seconds for this because I have some to go in class. I have five minutes left.

Okay, 5 seconds, guys. Let me end this part. So can anybody answer how many will be need?

Seven. Seven. Right. Six for the sides. Six sides and the seventh to them all. Right. Perfect. So that was that was for you know boolean outputs with boolean or real inputs. But then what if the output is continuous valued? How can we build a network for this? To see how this can be done, I'm going to consider a function of a scalar input. The input X is scalar. The output is also scalar. Now I can compose the simple net. The upper perceptron fires if the input exceeds threshold t1. The lower and lower lower one fires if it exceeds t2. So if I combine the two with weight one and minus1 respectively. Then as the input goes from minus infinity infinity to plus infinity. When it crosses t1 this guy fires but until it crosses t2 this doesn't fire. Right? So between t1 and t2 the output is one. But once the input exceeds T2, they cancel each other out.

So this network produces a one when the input lies between T1 and T2. And once I can build that, then if I have a function of this kind, I can always approximate this function in this pie-wise linear manner. And so I can have pairs of neurons for every little bar, scale their output by the height of that function within the bar and sum them up. And that's going to give me a pretty good approximation for this continuous value function. And I can make this function approximation arbitrarily precise by simply making these bars narrower and narrower. That make sense?

Does this did this approximation make sense? Yes or no? I have like one one yes. What about the rest of you guys? It's easy, right? And so I'll skip this poll, right? The story so far we've done, we've seen that MLPS are connectionist computational models and they're classification engines, but we just added to the story that they can also model continuous valued functions. And so MLPS can do a whole lot of other things. They can uh model memory. They can per they can model probability dist using uh neoly networks. They can model probability distributions over real complex and integer domains. They can model a posteriority and a priority probability distributions of data. They can even generate data from complicated or unknown probability distributions. So from the AI perspective, the neural network is just a function given one or more inputs. It computes the function to predict one or more outputs.

Going back to the beginning of class, in all of these tasks which we saw early in the lecture, these boxes are just functions. They take an input and produce an output. So the voice recognition system, for example, takes in voice and produces a transcription. The image captioner takes an image, it produces a caption. The game playing machine, it takes in a game state, it outputs the next move. So these are all just functions. And so when we do it with neural networks, all of these functions are modeled or approximated by neural networks. And so that's how we perform AI using neural networks. And so to the to to complete the story so far, interesting AI tasks are functions that can be modeled by the neural network.

And so in today's class, we've gone through a brief history of neural networks of connectionism. We've seen some early models and their limitations. We've introduced modern neural networks and what they can compute. And so here's where we are. MLPS can basically model any kind of computational task. They are universal approximators. But then there's more to it than that. And in the next lecture, we'll talk more about MLPS as universal approximators and the issue of depth in neural networks. So I'll stop here right on the dot and I can take questions.

Questions about this episode

What is discussed in "CMU Introduction to Deep Learning 11785, Spring 2026: Lecture 1"?

In order: 0:00 Neural Network, AutoLab, Kaggle; 10:00 Neural Network, Connectionism, Cognition, AutoLab; 20:00 Neural Network, Cognition, Google Translate, AlphGo; 30:00 Neural Network, Connectionism, Hebbian Learning; 40:00 Neural Network, Perceptron, Connectionism, Cognition; 50:00 Perceptron, Neural Network, Hebbian Learning; 1:00:00 Perceptron, Neural Network, Hebbian Learning; 1:10:00 Perceptron; 1:20:00 Neural Network, Connectionism.

What is mentioned in this Carnegie Mellon University Deep Learning episode?

12 people, works and ideas are mentioned, including Neural Network, Artificial Intelligence, Perceptron, Connectionism, Hebbian Learning, Cognition. Each one on this page links to its own page with every episode that mentions it.

How long is "CMU Introduction to Deep Learning 11785, Spring 2026: Lecture 1"?

The episode runs 1 h 23 min; the transcript is about 11,770 words in 110 timestamped sections.

More from Carnegie Mellon University Deep Learning

About this transcript: the episode's spoken words as captioned on YouTube, split into timestamped sections. Reclip.Me locates every person, work and idea in the text, keeps the time it is first named, and quotes the words around it verbatim. Last updated January 24, 2026.