Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Dylan Liu

What this episode is about

The presentation focuses on supervised learning techniques applied to the BirdCLEF 2025 competition, highlighting the speaker's self-taught journey in AI and machine learning, as well as their strategies that led to a successful outcome, including the use of custom soft loss and specific data normalization methods. Key points include insights into leveraging unlabelled data, hyperparameter tuning, and the impact of these approaches on improving leaderboard rankings in machine learning competitions.

Key takeaways

• Self-taught AI wizards: the internet's DIY school beats formal education for many aspiring data scientists.
• Kaggle's prize hunters thrive on competition; success is sweeter when it comes with a financial reward.
• SCD solutions are the new black—adopting trends from past contests can lead to surprising upswings.
• Overfitting woes? Switch to SED mode—stability might just save your model from meltdown.
• Soft labels aren't just fluff; optimizing relative probabilities transforms prediction accuracy like magic.

Who speaks in this episode?

• Dylan Liu (Host) · 10 min speaking

What is discussed, and when

1. 0:00 – 3:00: B Cliff, Kaggle
2. 3:00 – 6:00: efficient net, SED mode
3. 6:00 – 9:00: AOC loss
4. 9:00 – 11:37: efficient net

Notable quotes from "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Dylan Liu"

• So thank you for attending my presentation. The presentation title is supervised learning with software a laws for B cliff 2025.

  about B Cliff · at 0:00

• Competition to win the prizes actually I'm a this is my main income source I'm a kind of prize hunter on kegle drum data and top coder I participant in some audio…

  about Kaggle · at 0:53

• Use SED mode, the training is also SED mode training method. I found that the SED mode is more stable most way more stable than classification mode. the actually you try if you train the classification mode after about 10.

  about SED mode · at 4:17

• 16 efficient net modes as the final solution. only efficient net works. I don't know why I tried this net. I tried transformer. none of this worked.

  about efficient net · at 5:43

• Like cross entropy loss this loss will optimize the relative possibilities training with this loss will enlarge the difference between the positive predictions and the negative…

  about AOC loss · at 6:53

Who and what is mentioned in this episode?

Companies & orgs

• Kaggle — Company · Platform for competition · 1 mention · first at 0:59
  Platform where the speaker participates in machine learning competitions

Concepts & topics

• efficient net — Concept · Core topic · 5 mentions · first at 5:12
  Central to the speaker's solution in machine learning competitions, used as labeling and final solution modes
• SED mode — Concept · Strategy · 2 mentions · first at 4:14
  Training method used by the speaker, noted for its stability and resistance to overfitting
• AOC loss — Concept · Key argument · 2 mentions · first at 6:12
  Custom loss function developed by the speaker, crucial for improving competition ranking

Places & events

• B Cliff — Event · Referenced event · 3 mentions · first at 0:05
  Mentioned as a context for the speaker's presentation and solution inspiration

Full transcript with timestamps

0:00 So thank you for uh attending my presentation. The presentation title is supervised learning with software a laws for B cliff 2025. Uh this is a template template template. Uh I'm a self learner. Uh I learned all my AI or machine learning knowledge online from the internet. Uh yeah also of course I read some books but most of my knowledge was for on the internet. I learned not I learn I learned uh knowledge for uh skill for on the internet. I also work on internet. I'm a I'm a freelance program uh yeah do some AI related job jobs. uh recently in recent years I practiced uh uh participants participated in various machine learning data science competition to win the prizes actually I'm a this is my uh main income source I'm a kind of prize hunter on kegle drum data and top coder uh I participant in some audio classification competition uh the last uh the last audio uh audio classification competition Uh I won was

1:15 this this on data. This is this is actually is speech text matching competition but uh uh I I just uh treated it as uh audio classification on data. I was uh second place the code is uh uh open sourced and the summary uh I use the SCD solution uh inspired by the Bclave 2023 second play solution uh and also I used the custom soft loss which this is this is very important this helped me improve my place from the uh uh 11s on public leaderboard to fourth on uh private leaderboard. I will uh I will explain it later and the simp semir learning um yeah just uh to use all the unlabelled data like others solution uh only 2D batch normalization uh I actually I tried a lot of uh data normalization method but only 2D batch normalization from the uh B cliff 2023 thank the play solution worked Uh the training time was about uh 50 50 hours on uh 40 1490 including the 10 hours data processing part without need

2:52 of GPU GPU. Yeah. uh facial selection and engineering. Uh I did some hyperparameter turning uh and found out uh it seems like only these two parameters matter uh whole length smaller whole length and the larger mouse. uh other other hyperparameter parameter seems not seem not so important and the human speeds filtering a human this is this this is directly from uh uh public notebook and the frequency dropout frequency masking actually this is one part of the of my uh of my data augment augmentation part. But the list is the most important. Uh not only the high frequency masking but all but the full frequency masking and uh mix up both audio and the male spectrum and this is important size. Uh the important side that's less than this will cause drop on the leaderboard score.

4:11 uh training method sed mode training. Yeah. So uh since I use SED mode, the training is also SED mode training method. Uh I found that the SED mode is more stable uh most way more stable than classification mode. Uh the the actually you try uh you try if you train train the classification mode uh after about 10 uh epics you will get over fit to the training data set but you can try a lot of more epics uh uh the the sed mode it's more stable and also this sed mode uh will less sensitive to random uh custom soft loss soft loss uh which is it seems very resistant to overfitting uh supervisor learning to make use of unlabelled data uh 10 efficient net modes as the labing mode. Yeah, I firstly I I try 10 efficient modes from the small one to large one uh as the leveling mode and then use the lably mode to generate the uh studio software labels uh super sampling of uh uh classes with few samples that is real classes. This

5:39 is also the common common technique. Uh 16 16 efficient net modes as the final solution. Uh only efficient net works. I don't know why I tried uh this net. I tried transformer. Uh none of this worked. Uh next this is the uh the code of the my my my loss. Uh actually this is the the so-called AOC loss is not the same as AOC uh metric but uh it's it is for AOC metric. Uh you can see this this part this part is can you see the the mouse?

6:28 Yeah, we can see it. Okay. Okay. Um this part is is the soft soft level part. I use the concept of the sample wet to implement the soft labs and this is the loss part. You can say instead of directly optimizing the absolute absolute possibility uh like cross entropy loss this this loss will optimize the uh the relative possibilities training with this loss uh will enlarge the difference between the positive predictions and the negative predictions because AOC metric relies on relative possibility. is not absolute.

7:16 It relies on the kind of order of the of the possibilities. So important and interesting findings a small amount of unlabelled data from the past competition with soft labels of the uh sudo labels uh improved the mode slightly. I I was finding I was trying to find some additional data. I so I tried the unlabelled data from past competition. I selected a small amount of them based on the summary of the uh software labels. Uh it improved the most slightly but because it was very time consuming and uh uh and the the domain actually the domain the domain data domain of past competitions is is different from the current domain. So I I I didn't add the this method to my final solution.

8:16 Maybe next time I will try to find a way to use the the unlabelled data from the past competition. Uh delighting the predictions that means that they all to zero of some classes some real classes had little impact on readable score. It seems that my mode cannot efficiently predict this classes. I don't know if it's only me or uh other participants also face this problem. Uh this is the this is the example uh example of class 963335. Uh it this class has only one sample. So basic predictions this is the basic predictions. This is the basic predictions. uh but uh where all the predictions of this class uh were set to zero. You can see the result barely changed and uh also this is a bas predictions where the prediction of this class uh was set to one minus original uh predictions. that means the the the order of predictions uh is reversed.

9:36 So the the the result still barely changed. So my mode is are able to predict classes like this. I guess uh simple mode uh efficient uh yeah efficient net way 2s contributes the most. It has the highest weight in my final solution. If you want to uh get a faster solution, you can just remove remove some large modes. Yeah, it's simple because I just I use all the mode I you I used is efficient net from small one to large one and uh uh yeah and uh you can also reduce the size of the input image import feature because uh efficient net takes dynamic uh import size and I recommend you to return it uh after you change the the import size uh solution over over wave.

10:38 Uh biggest advantage is the custom soft loss uh and uh 16 efficient net modes is my final ensemble ensembling and uh most unlabelled data was used for semi-supressed learning. uh unlike unlike other some of other solutions I just uh I use almost all unlabelled data and add and generate the CU labels for them and post that to base training data as the one data data set uh I just cut the uh 10% of the unlabelled data with a few uh lower summary of the soft label at the CD levels. Okay, it's fewer time.

Questions about this episode

Who speaks in "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Dylan Liu"?

Dylan Liu (Host).

What is "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Dylan Liu" about?

The presentation focuses on supervised learning techniques applied to the BirdCLEF 2025 competition, highlighting the speaker's self-taught journey in AI and machine learning, as well as their strategies that led to a successful outcome, including the use of custom soft loss and specific data normalization methods. Key points include insights into leveraging unlabelled data, hyperparameter…

What is mentioned in this Kaggle episode?

5 people, works and ideas are mentioned, including efficient net, SED mode, AOC loss, Kaggle, B Cliff. Each one on this page links to its own page with every episode that mentions it.

How long is "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Dylan Liu"?

The episode runs 12 min; the transcript is about 1,370 words in 10 timestamped sections.

More from Kaggle

• All analyzed episodes
• People mentioned on Kaggle
• Topics mentioned on Kaggle