0.20 Losses part1

Who speaks in this episode?

• Mchun Jang (Main speaker)

What is discussed, and when

1. 0:00 – 5:00: Mchun Jang, PyTorch
2. 5:00 – 10:00: Mean Squared Error, PyTorch
3. 10:00 – 15:00: Cross Entropy Loss, binary cross entropy, Mean Squared Error, PyTorch
4. 15:00 – 20:00: center loss, Cross Entropy Loss
5. 20:00 – 25:00: Cross Entropy Loss, additive angular margin softmax, angular softmax
6. 25:00 – 27:10: triplet loss

Notable quotes from "0.20 Losses part1"

• Hello everyone. Welcome to recitation zero loss part one. My name is Mchun Jang. Today we are going to talk about loss functions which are actually what drives a model towards…

  about Mchun Jang · at 0:04

• Actual code. First before training we define the loss function usually called the criterium. This can be adding existing PyTorch loss or custom one you define yourself. Then we…

  about PyTorch · at 1:55

• Among this, the most commonly used loss for regression is mean square arrow or MSE. It can be interpreted as the average squared ukidian distance between predictions and targets over a batch of.

  about Mean Squared Error · at 8:45

• Reduces to the negative log of the predicted probability of the target clause. The goal of cross entropy loss is therefore to encourage high probability for the crack clause.…

  about Cross Entropy Loss · at 12:05

• Computed using py and y minus py instead of multiple class probabilities. As before the objective is to minimize the loss encouraging the predicted probability to match the…

  about binary cross entropy · at 13:34

• Now let's look at the first loss in the zeros shot family center loss. With center loss we explicitly try to shrink each clause by pulling the feature vector of each data point toward the center of its class. The loss is defined.

  about center loss · at 17:06

• Should be clearly separated while embedding of the same person should be very close. The objective is to maximize co similarity between a sample and its cracked class center…

  about angular softmax · at 21:11

• The objective is to maximize cosen similarity for the crack clause while enforcing an additive margin to separate clauses in the embedding space in the formula. This is done by.

  about additive angular margin softmax · at 23:58

Who and what is mentioned in this episode?

People

• Mchun Jang — Person · Main speaker · 1 mention · first at 0:04
  Introduces and teaches the session ('recitation zero loss part one'), explaining loss functions and how they are used in PyTorch training/evaluation loops.

Products & tech

• PyTorch — Product · Tool mentioned · 3 mentions · first at 2:03
  PyTorch is mentioned as a framework used for implementing loss functions in machine learning models.

Concepts & topics

• Cross Entropy Loss — Concept · Core topic · 7 mentions · first at 11:08
  Cross entropy loss is a fundamental concept discussed as a key loss function for classification problems in machine learning.
• Mean Squared Error — Concept · Core topic · 2 mentions · first at 9:29
  Mean square error is highlighted as a common loss function for regression tasks, measuring the average squared difference between predicted and actual values.
• binary cross entropy — Concept · Core topic · 3 mentions · first at 12:32
  Binary cross entropy is presented as a specialized form of cross entropy loss for binary classification problems.
• center loss — Concept · Core topic · 3 mentions · first at 17:08
  Center loss is discussed as a method to improve feature clustering in zero-shot learning by minimizing the distance between samples and their class centers.
• angular softmax — Concept · Core topic · 1 mention · first at 21:03
  Angular softmax is introduced as a margin-based loss that focuses on angles between feature vectors to improve class separation in zero-shot learning.
• additive angular margin softmax — Concept · Core topic · 1 mention · first at 22:33
  Additive angular margin softmax is described as a variant of angular softmax that enforces a fixed margin to improve class separation.
• triplet loss — Concept · Core topic · 1 mention · first at 25:40
  Triplet loss is explained as a contrastive loss that enforces relative distances between samples to improve feature separation in zero-shot learning.
  Is another image of your face and the negative is an image of a different person. The objective is to minimize the distance between the anchor and the positive while ensuring…

Full transcript with timestamps

0:04 Hello everyone. Welcome to recitation zero loss part one. My name is Mchun Jang. Today we are going to talk about loss functions which are actually what drives a model towards success. In this restation we'll cover what a loss function is, why it is so important and briefly introduce different types of losses. More advanced losses for specific tasks will be discussed in loss part two. So let's think about training the model as driving a car toward a destination. But how does the car know where to go and how to crack itself when it goes off track. This is where the loss function comes in. The loss function acts like the steering wheel well. It tell the model how far it is from the truth, mirrors how wrong the prediction is and guides the model to adjust during training. Without a loss function, training will be like driving without a steering wheel. You have no way to reach the destination. So what is a loss

1:06 function? It is a metric that measures the performance of a model. More specifically, it quantifies the arrow between the model's predicted output and the corresponding ground truth target. Why is it so important? Because the loss function tells us how different the model's predictions are from the actual answers. During training, this information is used to update the model's weights and gradually improve its predictions. In short, we need loss function for two main reasons. First to evaluate how well the model is performing and second to guide the learning process so the model can make better predictions over time.

1:53 So let's see where the loss is used in actual code. First before training we define the loss function usually called the criterium. This can be adding existing PyTorch loss or custom one you define yourself. Then we move into the training loop. We set the model to training mode loop through the data set. Run the forward pass to get predictions and then compute the loss using the model outputs and the targets. After that we call loss.backward and optimizer step to update the model parameters. For evaluation the process is similar but with one key difference. We switch the model to evaluation mode and do not compute gradients. Here the loss is only used to measure performance, not to update the model.

2:48 Now let's talk about the types of losses. In practice, loss functions can be classified in several different ways. At the most basic level, the most common and intuitive distinction is between regression losses and classification losses. depending on whether the model predicts continuous values or discrete classes. Losses can also be grouped based on the specific tasks they are designed for such as classification, ranking or generative modeling. We won't focus on these task based categories here and we'll introduce some of them in last part two.

3:25 For now, it's more helpful to think about losses from another perspective. Different losses optimize the model in fundamentally different ways. Meaning they provide different kinds of learning signals. From this perspective, losses can be broadly divided into three categories. First, arrow based losses. These losses directly measure the magnitude of prediction arrows. That is the difference between the model's output and the ground truth target. And training simply minimizes this arrow. Typical examples include MSE, MAE, and Huber loss, which we'll discuss later.

4:06 Second, probabilistic losses. Unlike arrow based losses that compare values directly, probabilistic losses treat the model output as a probability distribution and minimize a distance or divergence between the predicted distribution and the true lab true data distribution. These are commonly used in classification problems and we'll see them again later. Third, margin based losses. Rather than modeling absolute arrows or probabilities, these losses focus on relative separation enforcing a margin that pushes cracked predictions away from incorrect ones. Examples include contrastive loss, angular softax. We'll go into this family of losses in more detail in the later part of this restation.

4:59 So far, we've discussed loss functions based on their underlying principles or optimization strategies. But for this restation, we'll take a simpler and more practical viewpoint that better reflects how these losses are used in practice and that you also encounter directly in your homeworks. We will focus on two main types of losses. Basic losses and zero shaw losses. This distinction is about what kind of generalization behavior we want the model to learn. Basic losses are used when the goal is to perform well on the same set of classes or targets seen during training.

5:41 Zero shot losses on the other hand are designed for settings where the model must generalize to unseen classes and therefore encourage the model to learn more structured and transferable representations rather than just the fitting decision boundaries. Many of the margin based losses we just mentioned fall into this category. For example, in face verification, you don't want a model that only works well on a fixed training set of people. You want it to generalize to new identities that were never seen during training.

6:18 Otherwise, you would need to retrain the model every time a new person appears. You experiment with many of these losses yourself in homework 2b2. So, let's go over them in detail. So for basic losses, the first one we'll talk about is distancebased loss for regression. In regression model, we want the network's prediction to be as close as possible to the target value. Here, this is the network output and this is a target. So the goal is to minimize the distance between the prediction and the ground truth. But how do we measure this distance? There are different ways to define the distance between a prediction and a target. The simplest one is a direct difference which is just the prediction minus the target. This keeps a sign. So it tells us whether the prediction is above or below the target.

7:16 However, positive and negative arrows can cancel out which is why this is not suitable as a loss by itself. To address this issue, a more commonly used distance is the square distance, also known as L2 arrow. This distance is always non-active and from the formula you can see that it penalizes large arrows much more heavily than small one. As a result, it is more sensitive to out layers. You can also see the effect visually. The faded line tends to tilt toward our layers to reduce these large square arrows. Another option is absolute distance or L1 arrow. This penalizes arrows linearly and is therefore more robust to outlers.

8:05 However, the downside is that L1 loss is not smooth which makes the optimization harder. In practice, this means we need to rely on subgradients which is less convenient for gradient based optimization. Now let's move from distance to loss. Instead of looking at the distance for a single data point, we compute the distance for all samples in a batch and then take an average. This give us loss functions such as MBE, MSD and ME. There are also losses like Huber loss which are designed to balance the advantages of both L1 and L2.

8:45 Among this, the most commonly used loss for regression is mean square arrow or MSE. It can be interpreted as the average squared ukidian distance between predictions and targets over a batch of training data. So imagine you are building a real estate model that predicts house prices based on features like size, location, and number of rooms. In this case, a natural choice of loss is mean square arrow. From the figure, you can see the true house prices and the predicted prices. the square difference between them corresponding to the prediction arrow that the model is trying to minimize during training.

9:29 This is the formula for MSE. Here n is the number of data points, yhat is a predicted value and y is a ground truth value. For each data point, we take the squared difference between the predicted price and the true price and then average over all samples.

9:50 [snorts] Okay, the idea of MSE in code is very simple. First, we initialize the loss function. Using PyTorch, we can replace it with any loss we want. Here we use MSE loss. Then we pass in the model predictions and the ground truth targets. These are just dummy values, but you can think of them as real house prices and predicted prices. PyTorch then computes a mean square arrow for us. One very important thing is that the lower the mean square arrow, the better the model. When the loss value gets closer to zero, it means the predictions are closer to the ground truth.

10:35 Now let's move on to the second type of losses, classification losses. In a classification problem, the model outputs a probability for each class. These probabilities are usually computed using a softmax function which takes the row outputs of the network and normalizes them so that they are all non negative and sum to one to generate the probability. Ideally the probability for the crack class should be very high close to one while the probability for all other classes should be low. This behavior is captured by the cross entropy loss. One important thing to note is that the cross entropy loss is zero when the model assigns probability one to the crack class and greater than zero otherwise. Now suppose we are classifying images into categories like cat, dog or bird. In the formula yc is a ground truth label for class c. It equals one for the crack class and zero for all others. Because of this when we

11:41 multiply yc with log yc hat only the term for the true class contributes to the loss. The sum over classes means we consider all possible classes and the sum over data points means we average over the samples used to compute the loss which in practice is a current mini batch. So for a single sample the loss reduces to the negative log of the predicted probability of the target clause. The goal of cross entropy loss is therefore to encourage high probability for the crack clause. For example, if the true class is cat, the model should assign cat a high probability and dog and bird low probabilities.

12:29 Okay, here's a special case of cross entropy, binary cross entropy. In some classification problem, the model only needs to identify a single class answering a yes or no question. For example, given an image of a leaf, we want to know whether it is infected by fungus or not. In this case, the network outputs one probability, py for the target class. This value should be one for positive examples such as an infected leaf and zero for negative examples such as healthy leaf. Even though there is only one output, there are implicitly two classes. the probability of the positive class PY and the probability of the negative class 1 minus PY. Binary cross entropy is simply the cross entropy loss applied to this binary setting where the loss is computed over both possibilities. The formula is the same in spread as cross entropy but it is computed using py and y minus py instead of multiple class probabilities. As

13:40 before the objective is to minimize the loss encouraging the predicted probability to match the true label in code. Cross entropy and binary cross entropy work in a very similar way to what we saw before. We provide the true class labels and the model's predicted probabilities and PyTorch computes the loss for us. Here we initialize some dummy labels and predicted probabilities. In practice, this predicted probabilities usually come directly from the model. Then we initialize the loss function and pass in the predictions and true labels to compute the cross entropy loss.

14:24 So far we've talked about regression losses and classification losses. Both of these belong to basic losses and they are widely used across many deep learning problems. Next we'll talk about zerosaw losses. Here it does not refer to one specific type of loss function but to a learning setting where the model is expected to generalize beyond the classes seen during training. In the classification losses we've seen so far. The goal is to predict specific classes in the training data. Essentially maximizing classification accuracy of known classes.

15:06 In zero shellot setting, the goal is different. Instead of learning class labels directly, we want the model to learn a structured representation space where samples from the same class are close together and samples from different classes are far apart. The training classes are therefore only examples, not the final objective. Zero shot losses are designed to encourage this kind of representation learning. so that the same structure can generalize to unseen classes. So what is the difference between conventional classification and zeros shock learning? In conventional classification, the network mainly learns a decision boundary in the feature space. As long as different classes can be separated by this boundary, the loss is minimized even if the features themselves do not form meaningful clusters.

16:05 In zero shot learning, this is not enough because new classes appear at test time. The decision boundaries learned from training classes may not apply. As a result, we explicitly optimize for intro class compactness and interclass separation so that the representation space itself generalizes. One important thing to note is that cross entropy loss can sometimes behave like a zero shell loss. When trained on a large number of classes, cross entropy encourages features to become linear separable across many directions which can implicitly lead to clustering.

16:47 However, this clustering in fact is implicit and data dependent and often weak when the number of training classes is limited. So uh that's why more explicit zero shot losses are needed in many applications. Now let's look at the first loss in the zeros shot family center loss. With center loss we explicitly try to shrink each clause by pulling the feature vector of each data point toward the center of its class. The loss is defined as the total square distance between each sample and the center of its corresponding class. Importantly, the class centers are also learned during training. They are learnable parameters, not fixed values. Center loss is not used alone by itself. It does not enforce class separation. So, it is usually combined with cross entropy loss. Cross entropy helps separate different classes while cents makes samples within the same class more compact. For example, in a face

17:57 recognition system, CentOS is used together with cross entropy to improve performance. The model learns face embedding so that images of the same person are grouped closely around the class center in the embedding space. The objective is to minimize the distance between feature vectors of samples and their class center. In the formula, M is the number of samples in the batch. X I is the feature vector of the I sample and C YI is the center of class Y. To implement this in code, first we set the number of classes and the feature dimension. Then since the class centers are learnable parameters, so we initialize self dot centers as a n parameter starting from a random initialization.

18:50 In the forward function, we first pick the center for each sample based on its label. Then we compute the square distance between the feature vectors and their corresponding centers. After that we clamp the distances to keep them in a reasonable range and finally take the mean distance to get the loss. Here we use some dummy embedding vectors and their corresponding class labels. We initialize the center loss module with a number of classes and feature dimension. Then call it on the embodings and labels to compute the center loss.

19:30 After center loss, the next group is angular losses. They are widely used in tasks like face recognition, face verification and speaker verification where strong generalization to unseen identities is required. Angular losses normalize feature vectors to line on the surface of a sphere. So all vectors have the same length. As a result, angular losses focus on angles instead of distances. and training works by minimizing the angle between each sample and its class center. This is because feature magnitude is not reliable for representing class identity.

20:11 From the dot product, we know similarity depends on both magnitude and angle. Without normalization, the model could increase similarity simply by making feature vectors longer without learning better representations. But what we actually want is class level similarity. Samples from the same class could point in similar directions and samples from different classes should point in different directions. By normalizing the vectors, angular losses removes the magnitude term. So similarity depends only on the angle which better capture this structure.

20:52 This also make it easy to introduce an angular margin. adding actual penalty when the sample's direction is not close enough to its claw center. A representative example is angular softmax. Consider face recognition system embeddings of different people should be clearly separated while embedding of the same person should be very close. The objective is to maximize co similarity between a sample and its cracked class center while enforcing a margin between different clauses. In the formula, theta is the angle between the embedding and the crack class center. M is the angular margin that enforces stronger separation between clauses and s is a scaling factor used to stabilize gradients. These angular similarities are passed through softmax to form class probabilities and the minus log turns them into a loss that penalizes low probabilities on the crack class.

21:58 In code we initialize the margin scale and the class width. Here the class width W are learnable parameters that represent the class centers in the embedding space. We normalize both features and class weights. Compute cosine similarity. Subtract the angular margin for the true class and then apply scaling before using cross entropy loss. Here we use simple 2D embodings and their corresponding class labels as an example to compute the angular loss. Okay. Another type is additive angular margin softmax. This loss makes classes more distinct by adding an angular margin to the decision boundaries.

22:45 Compared to the standard softmax, this margin makes classification harder. So the model has to be more confident to classify a sample correctly. In standard softmax, the decision boundary is relative loose. With an additive margin, the boundary becomes stricter. Samples from the same class are pulled closer together while samples from different classes are pushed farther apart. This help improve class separation especially for challenging or similar classes. An important difference to note is that angular softax introduces an unfixed angular margin while additive margin softmax uses a fixed and explicit angular margin.

23:34 This fixed margin makes the separation effect more stable and controllable. A typical example is speaker verification. The goal is to determine whether two speech samples come from the same person. Embedding from the same speaker should be compact while embeddings from different speakers should be well separated. The objective is to maximize cosen similarity for the crack clause while enforcing an additive margin to separate clauses in the embedding space in the formula. This is done by subtracting the margin from the cosine similarity of the true clause compared to the standard soft max where no such margin is applied in code. This implementation is very similar to the previous angular loss.

24:29 The key difference is that we explicitly subtract the additive angular margin from the cosine similarity before computing the loss. Here we again use simple 2D embedding and their class labels as an example. Now let's move on to contrastive losses. The losses we've seen so far mainly focus on the relationship between each data point and its class. So the loss for each sample can be computed individually. In contrast, contrastive losses are computed by comparing data points to each other. The basic idea is simple. A data point should be closer to samples from the same class than to samples from different classes.

25:20 In the figure, the red dashed line represents the distance between two points from the same clause while the black dashed line represents the distance between points from different clauses. Ideally, the red distance should be always be smaller than the black one. One of the most common contrastive losses is triplet loss. Consider face recognition example such as unlocking a phone with your face. The anchor is the stored image of your face. The positive is another image of your face and the negative is an image of a different person. The objective is to minimize the distance between the anchor and the positive while ensuring that the anchor is farther from the negative by at least a margin ARA.

26:13 In the formula, we compare the distance between anchor positive and anchor negative. The margin enforces that the negative sample is sufficiently far away. A relu is applied here so that the loss is always non- negative. In code, we initialize the anchor positive and negative embeddings. Compute piewise distances and then compute the tripler loss at the positive distance minus the negative distance plus the margin. These contrastive losses are widely used in tasks like face recognition and verification where relative similarity between samples is more important than predicting a fixed cloth label.

27:03 Okay, that's all for this part. Thank you all for listening.

Questions about this episode

Who speaks in "0.20 Losses part1"?

Mchun Jang (Main speaker).

What is discussed in "0.20 Losses part1"?

In order: 0:00 Mchun Jang, PyTorch; 5:00 Mean Squared Error, PyTorch; 10:00 Cross Entropy Loss, binary cross entropy, Mean Squared Error, PyTorch; 15:00 center loss, Cross Entropy Loss; 20:00 Cross Entropy Loss, additive angular margin softmax, angular softmax; 25:00 triplet loss.

What is mentioned in this Carnegie Mellon University Deep Learning episode?

9 people, works and ideas are mentioned, including Cross Entropy Loss, Mchun Jang, PyTorch, Mean Squared Error, binary cross entropy, center loss. Each one on this page links to its own page with every episode that mentions it.

How long is "0.20 Losses part1"?

The episode runs 27 min; the transcript is about 3,601 words in 34 timestamped sections.

More from Carnegie Mellon University Deep Learning

• All analyzed episodes
• People mentioned on Carnegie Mellon University Deep Learning
• Topics mentioned on Carnegie Mellon University Deep Learning