Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Nikita Babych

What this episode is about

The presentation by Nikita Babych outlines his journey in data science, highlighting his experiences and strategies for winning the BirdCLEF 2025 competition, including challenges posed by imbalanced training data and strategies to address them. Key points include his background, the significance of tackling data imbalance across taxonomy groups, and the impact of noisy non-bird samples on the competition results.

Key takeaways

• Data imbalance can turn a competition into a shadow play; non-bird groups represented just 4% of the samples.
• Normalization isn't just a step; it's the lifebuoy for drowning samples—vital for accurate label mapping.
• Context is king: extending audio chunks from 5 to 20 seconds transformed indistinguishable calls into recognizable voices.
• Who knew? A mere nine non-bird samples make for a tough day at the data playground!
• Winning isn't about flashy models; sometimes it’s about sticking to proven setups that actually work.

Who speaks in this episode?

• Nikita Babych (Host) · 22 min speaking

What is discussed, and when

1. 0:00 – 5:00: Kaggle, Zbear
2. 5:00 – 10:00: self training, noisy student, CNN, GM frequency pooling
3. 10:00 – 15:00: self training, noisy student, power transformation
4. 15:00 – 20:00: self training
5. 20:00 – 24:04: self training, efficient net b 0

Notable quotes from "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Nikita Babych"

• Like generative and at light image m yeah and u for the past one and a half years I've been a kaggle addict it means that I've been participated in multiple competitions…

  about Kaggle · at 0:56

• Bit slightly better results with cross entropy and I my intuition behind that is that nature of soft max that lies behind cross entropy helps a bit to deal in certain cases with class imbalances.

  about Cross Entropy Loss · at 7:03

• Mixups. Otherwise mix up didn't make much sense. yeah talking about validation I didn't manage to find a reliable correlation between local cross validation and the bear. So that's why I made a decision to use only zbear as my.

  about Zbear · at 8:06

• Next improvements should come with the use of unlabelled data. So that's why I went through multiple papers on self training solutions of the last year and I noticed cu a…

  about self training · at 9:17

• Noisy and I had to reduce noise by applying some transformation and surprisingly the power transformation turned out to be the greatest way to reduce that noise. So on the…

  about power transformation · at 14:19

• Spot of them. so I had to train model for 40 epox larger batch size and smaller model like efficient net b 0. I was really surprised that I couldn't improve results training…

  about efficient net b 0 · at 20:08

Who and what is mentioned in this episode?

Companies & orgs

• Kaggle — Company · Platform · 2 mentions · first at 0:00
  Mentioned as the platform where the speaker participated in competitions and gained knowledge.

Concepts & topics

• self training — Concept · Strategy · 8 mentions · first at 9:22
  A machine learning strategy discussed as a method to improve model performance using unlabelled data.
• noisy student — Concept · Methodology · 3 mentions · first at 9:35
  A methodology applied in self-training to improve model performance by injecting noise.
  Went through multiple papers on self training solutions of the last year and I noticed cu a certain tendency that self training starts work starts working in our domain when we…
• Zbear — Concept · Validation Metric · 1 mention · first at 3:50
  Used as a validation metric to assess the performance of the speaker's models.
• Cross Entropy Loss — Concept · Loss Function · 4 mentions
  Used as a loss function during supervised training, chosen over others for better results with class imbalances.
• CNN — Concept · Model Architecture · 1 mention · first at 6:34
  A foundational architecture used across different training stages in the speaker's solution.
  The second place sorry from the first place 2021 GM frequency pooling and three families of CNN. The exact list of models you can see on your screen. yeah so about models…
• power transformation — Concept · Preprocessing Technique · 1 mention · first at 14:25
  Applied to reduce noise in labels during multi-iterative self-training, improving model convergence.
• GM frequency pooling — Concept · Model Component · 1 mention · first at 6:29
  Used as part of the model architecture setup, contributing to the speaker's solution.
  Architectures. So across different training stages I used the same setup that consisted of a stat head from the second place sorry from the first place 2021 GM frequency…
• efficient net b 0 — Concept · Model Architecture · 1 mention · first at 20:14
  A smaller model used for specialized classification tasks within the speaker's solution.

Full transcript with timestamps

0:00 Well, um first of all uh thank you Kaggle. Thank you dear listeners for your attention. Thank you for the opportunity for the opportunity to present my solution here today for you. Yes. So without further ado, let's get started. Now on your screens you can see the outline of uh my presentation and the first thing that I want to share with you a bit is about my background. So I'm 25. I'm from Ukraine. I have a bachelor's degree in physics. Yeah. And graduating university by chance I fell in love with data science. So I started studying it on my own and then within a year I got an offer from uh quantum.

0:41 It's an Ukrainian outsourcing company in keep working in my first year. Now um working there I managed to participate in multiple projects from absolutely different data science fields like generative and at light image m yeah and u for the past one and a half years I've been a kaggle addict it means that I've been participated in multiple competitions without long poses yeah and that's exactly how I managed to accumulate enough knowledge to secure the first place in this uh year bird clap. Um yeah, so having enough about me, we can start discussing my solution.

1:23 Uh as always, everything starts with data. So here I highlighted the uh most noticeable challenges that training data gave us. Uh first of all, we had a huge imbalance among taxonomy groups. Uh on the right you can see histograms with number of samples for each group. And very interesting here think here that non- bird groups are even hardly noticeable. Yeah. And to be more specific uh despite making up 30% of all species uh they uh made up less than 4% of total samples. Yeah. And uh to make these challenges even more fun uh non-bird groups samples with them was super noisy. Yeah. So from the beginning I figured that if I want to show uh somehow competitive results I need to deal with that uh challenges and the first thing that came to my mind of course was to uh check the portal for additional data. Um but unfortunately Zento didn't have a lot of samples for non-bird groups. Uh to be more specific

2:25 just nine samples. Uh yeah but with birth sation was a bit better but as you remember births won't our main issue this year. uh that's why uh incorporating the data into the training process I didn't see any noticeable improvements uh yeah and uh the final assemble uh included only one model that saw that data and that's what was done only for the sake of diversity yeah but I had a bit more fortune with uh non-target species uh for insect and amphibia groups then had a lot of samples for them so I downloaded about 17,000 uh samples. Yeah, initially I wasn't sure how I was going to use this data.

3:10 But then I found the way and I'll explain that way a bit later because now we need to discuss a bit more important matters like data prep-processing. Uh here I highlighted the most significant steps that affected results mostly on Zbear. First of all, I had to normalize row signals by absolute maximum. Uh it was absolutely necessary because uh samples varied a lot in terms of volume and mixing up uh samples of different volume sometimes resulted that the more quiet sample just disappeared and as a as a a result I got broken labels and that diminished uh my result on Zbear. So that normalization was quite important for me. Yeah. the next step that I had to set all secondary labels to one. Uh that worked for me because I use relatively long input channel durations and the chance to catch secondary spees was relatively high with them. That's why I got better results when I set sing labels to one. And yeah, to be more

4:14 specific about chunk duration, uh I use 20 second chunks. And to give you uh to share the motivation that I had behind that, I prepared uh some visualization of uh spectrograms. So here you can see uh a train uh soundsscape uh it's specttogram and you can see that in the red I highlighted the central p 5 5-second uh area of the signal that goes through the entire spectrogram and to emphasize the problem I highlighted in the blue rectangle another uh repetitive signal but only in the short term on the left and as you can see uh model don't have enough context to distinguish these signals Well, uh even if this specific one are quite distinct distinctive on their own, but there were definitely more similar signals and uh long uh sorry the short term wasn't enough to uh give model opportunity to learn uh relevant features to distinguish them.

5:12 Well, so extending that window uh to 10 seconds uh besides getting noticeable improvements on the B. Yeah, I've got a kind of logical evidence that now at least for these two species model had enough of the context to distinguish them well. But definitely there were even longer signals uh and specious calls like one that go through the entire specttogram here. And to distinguish them I it was absolutely necessary to show model even more context. Uh yeah, so I kept in extending video up to 20 seconds. And here I got the optimal value that worked for me best. Uh yeah, and any further uh attempt to extend the time video didn't give me any improvements that why I settled with that value. And my hypothesis is that that value should be tailored specifically for each new set of species because pecularities of their calls in my opinion almost fully determine that optimal value.

6:13 Um yeah talking about models to be honest I didn't uh come up with any breakthrough in terms of model architectures. So across different training stages I used uh uh the same setup that consisted of a stat head from the second place uh sorry from the first place 2021 GM frequency pooling and three families of CNN. The exact uh list of models you can see on your screen. Um yeah so about models that's it. Um yeah started I with the supervised training of course means that uh during that stage uh models saw only training data about important rating details uh that I used cross entropy instead of more common uh losses like uh binary cross entropy or focal loss uh because I got a bit slightly better results with cross entropy and uh I my intuition behind that is that nature of soft max that lies behind cross entropy helps a bit to deal in certain cases with class imbalances.

7:17 Yeah, talking about scheduleuler, I used co scheduleuler with warm restarts after each fifth epoch. Uh it uh helped me to train model for more epox uh to be more specific for supervised training it was 15 epochs. Uh while uh training with one cycle model started overfeitting already after 10 epox. Yes. So uh warmer stars worked great for me. Uh talking about augmentation surprisingly only one augmentation worked for me. It was mix up. Uh vical sampling weight for each species in order to deal better with uh class label imbalances. Uh yeah and the last but not least I used left padding strategy. It means that I put all samples on the right uh in order to get a consistent overlap between them after mixups. Otherwise mix up didn't make much sense. Um yeah talking about validation I didn't manage to find a reliable correlation between local cross validation and the bear. So that's why I made I made a

8:21 decision to use only zbear as my validation and then this time hosts said that they were expecting to see very similar distribution between public and uh private. So I calmed down and just kept using that validation. Uh and one more thing about validation that uh submitting single models uh I got uh noticeable fluctuations on the uh elbear that's why it was important to train and sample of models with the same setup I mean uh but with different fs or seats and in that way I got more stable results and that's that was enough for me to at least understand whether my my ideas worked or didn't um um yeah hitting the ceiling with uh uh supervised training. I figured that uh next improvements should come with the use of unlabelled data. So that's why I went through multiple papers on self training solutions of the last year and I noticed cu a certain tendency that self training starts work uh starts

9:31 working in our domain when we inject some kind of noise into the training process. So I related that to the noisy student approach and started to following it and it showed good results for me. Uh essentially noisy student can be boiled down to two main points. Uh first of all uh we need to include training data into soft training because it uh very beneficial especially during the uh the first uh training efforts. Uh yeah and the second uh uh point is that we need to inject some kind of noise that's why noisy student uh that noise can be split into two types of noise.

10:08 the first type that we need to augment strongly set labelled samples like input to the model and the second noise it's model level noise like uh dropout. So having that conception in the mind I went through different combinations of noise and came to the combination that uh actually enabled self training for me. uh that combination consisted of first of all mix up uh uh uh which was done by uh mixing up training samples with random Seattle samples that approach was used with some variations in the second place 2024. So I'm really grateful uh to the author of that write up because it inspired me to a lot of things. Uh yeah. So on the right you can see an example of that mix up. When we took from the left passive soundscape, from the right focal train uh soundscape, we mixed up them and as a result we get an absolutely win-win situation because for the left we inject necessary noise to enable self training

11:15 but for the right we inject uh real test domain background with potential labels in that background. So as you can see that was really great augmentation for self training and it had a lot of benefits from absolutely each side. Yeah. And another thing that significantly improved my results during self training it was the use of stoastic depths. In other words, dropout on residual blocks. Uh yeah. Uh so it was in line with the noisy student methodology. Uh and it didn't work during the supervised training stage. So but for self training it uh really helped me to get much more stable results in improve them.

12:00 Um about the mentioned earlier mixup uh it had very important parameter the ratio of mixups it means how many train samples dur in each batch were mixed up with random cell labelled samples. I say that it's important parameter because minor adjustments of that parameter resulted in significant fluctuations on the LB. So I conducted multiple experiments with different values and got to the following empirical results that the higher the ratio the higher the LB score. On the right you can see a plot where I incrementally was increasing the ratio of mixups and was checking the best score and got an clear positive correlation between these two variables. So turned out that the best ratio was one meaning that each training sample is always mixed up with obser label sample. Uh so setting that value for all training stages I kept training my models.

13:00 Uh about training details nothing significantly changed thin supervised training stage uh with a few exceptions. Uh first of all I had to train my models for more ports uh 25. seconds that I had to activate drop pass rate. Uh in other words, stoastic depths and ping strategy was switched from uh the left uh strategy to the random ping strategy because now I could place short training samples at random position and had consistent overlap with soundscapes and it a bit improved my uh results. Yep. Uh so being unable to get more from a single sub training iteration uh I uh became inspired by uh potential that I saw in multiple papers and solutions of the last years that showed that if we uh apply the same subranian approach multiple times by regenerating t labels we can get uh uh much more from unlabelled data and pair the labels. So I started experimenting with it. Uh unfortunately I didn't good I didn't get

14:10 uh good results uh uh firstly uh but then I investigated the labels that I got and I figured that they became too noisy and I had to uh reduce uh noise by applying some transformation and surprisingly the power transformation turned out to be the greatest way to reduce that noise. So on the right on the left you can see an example of that when initially I had a lot of noise in blue. Yeah. But then I applied uh the power transformation and uh all that noise just disappeared. Of course uh some overconfident labels became reduced. Uh that were probably true positives but empirical results showed that uh their values were still enough to let model converge uh well uh on them. Um yeah um another thing was that I had to use a sampler because not all uh samples were labeled equally well. Uh that's why uh based on the assumption that ideal lab should result in higher sum of s labels because listening to

15:18 multiple temples I really noticed that almost each soundsscape had a lot of species there inside. Uh yes. So that's why I uh assigned the weight sequel to the sum of C labels to the sampler. Yeah. And uh that worked great for me and uh most important it made my results much more stable. Uh okay. Uh so moving on here you can see the final uh multiative self-raining loop that starts with the generation of the labels for the first uh selfraining iteration. uh set of labels was generated by the best insemble from the supervised training stage and for later iteration was done by best models from the previous iterations. So uh having observed the labels it goes on to another inner uh loop the purpose of which is to tune the power transform value because results was super super sensitive to its value and uh usually it took multiple iterations to find the value that showed the best results. So I

16:29 had a hypothesis set power transform value calculated sampler weights uh applied self training uh that was described earlier uh subitting result uh results got the band feedback and decided what to do next. If I uh understood uh that I could get more from the current iteration I kept looping uh that uh circle. Yeah. But if I understood that I can get uh more from that iteration, I just broke from uh that loop and moved on to uh another station where I reset um current teacher model with a new one by selecting the best ensemble from the current iteration. Uh yeah, and then use that ensemble to generate regenerate the labels for the next iteration. Yeah. uh and uh that Yeah. Yep. Listening.

17:27 I was just curious. Um it's really nice seeing that sort of convergence on the public leaderboard scores. So I was wondering if you tried more iterations and it just leveled out at 0.93 or if it dropped off or if you only ever tried four. Yeah, just was about to say about that that I uh uh repeated that loop four times and uh I stopped getting any improvements after fifths after reaching first epoch because uh just result remains the same all the time. Uh no matter uh uh no matter what I tried to adjust in terms of parameters. Yeah, I tried to replace uh transformation that I used to reduce uh the level of noise.

18:09 I tweaked different uh parameters like number of epox and so on and just nothing worked for me. Uh yeah my intuition behind that that everything uh has its own limit especially the use of unlimited data because infinite engine doesn't exist and it would be really strange if we could get 100% of accuracy using several labels. Yeah. uh lucky labelers are they are still in huge demand for us data scientists. Um so uh moving on uh yeah I want to talk about a bit about a different uh branch of my solution. It was a dedicated model for insect and amphibia groups. Uh the motivation behind that was that again I had an assumption that almost each sunscape had a lot of species from insect and ania groups. I was listening and I definitely heard uh species from insectctor and anibia and then I was uh even more surprised understanding that training data was so imbalanced towards these groups because in sector made up

19:20 just uh 0.5% and amphibia 2% of data and even this data was very noisy. So it became clear that we need uh a model that would specialize kind of better in these groups and learn features specific uh for these groups. Uh that's why I decided to follow approaches of the uh past years. Uh but that was uh applied for birds. I just united all training data and additional data denited their labels and got in total 700 uh labels. Yeah. and train model to classify all these labels. Uh I have to say that that model was very sensitive to training parameters. So it took multiple iterations to find the kind of sweet spot of them. Uh so I had to train model for 40 epox larger batch size and smaller model like efficient net b 0. I was really surprised that I couldn't uh improve results uh training deeper models. But to be honest, I think I just didn't manage to find that sweet spot of parameters for deeper models and uh it's

20:31 possible to get uh better results. Yep. Yep. Um talking about Yeah. And for most uh attentive listeners, I want to share small validation trick again that's based on the assumption that train data should be trained should be populated mostly with target species. That's why I calculated average probabilities and checked that target species had a higher average probabilities than non-target species. So local validation showed similarly good results. But uh checking this distributions I could reject submitting and wasting precious precious daily submissions for uh distributions like here above. Yes. So 20 epox such distribution 40 epox such distribution uh and uh target species are dominant here. So I I decided to submit and usually got good results.

21:25 Uh yeah and uh having enough about training uh I want to share my uh inference approach. I used 1B sliding video uh segmentation. Uh it means that I was averaging overlap and pre-wise predictions from neighboring audio chunks then flattened all this pre-wise predictions into one long vector and then applied max cooling in order to get the expected number of predictions. In our case it was 12 that had uh multiple benefits. First of all I made use of all predictions. Uh the second that I uh considered this as a form of TK because each frame wise predictions was the result of assembling multiple ensemble predictions for the same frame but by inferencing uh chunks with a different surrounding context around uh that frame. Yeah. From other inference tricks I applied smoothing well known blending of uh predictions from neighboring chunks and delta sheet. You can check the second place solution 2023 for more

22:29 information about that. Uh yeah, so my final assemble consisted of seven seven models written diverse data set and self training iterations. the exact list of models you can see under the assembler breakdown with their specifications but I want to just summarize it saying that the key to survival shake up in my opinion was assembling models from different training stages models with diverse backbone architectures and models for the under represented taxonomic uh groups. Yeah. Now as a form of conclusion I just want to highlight most important findings. First of all uh I need I needed to uh tailor ch duration for the target species. Second that selftraining becomes effective when noise is injected in a domain relevant way specifically for the approach that I uh was using.

23:19 Next that multi iterative selftraing unlocks more value from unlabelled data means that usually you can't get maximum from self training approach that you chose uh using uh training only for one iteration. And uh the last but not least that usually all this self training doesn't work out of box and you have to go through multiple uh uh tuning and uh finding of appropriate preprocessing uh functions in order to reduce your loss that tend to be accumulated during self training and the smart sampling strategy that usually is based on your assumptions about how uh good labeling should look like in terms of uh distribution. Um yeah, so thank you for your attention, guys.

Questions about this episode

Who speaks in "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Nikita Babych"?

Nikita Babych (Host).

What is "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Nikita Babych" about?

The presentation by Nikita Babych outlines his journey in data science, highlighting his experiences and strategies for winning the BirdCLEF 2025 competition, including challenges posed by imbalanced training data and strategies to address them. Key points include his background, the significance of tackling data imbalance across taxonomy groups, and the impact of noisy non-bird samples on the…

What is mentioned in this Kaggle episode?

9 people, works and ideas are mentioned, including self training, noisy student, Zbear, Cross Entropy Loss, Kaggle, CNN. Each one on this page links to its own page with every episode that mentions it.

How long is "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Nikita Babych"?

The episode runs 24 min; the transcript is about 3,624 words in 25 timestamped sections.

More from Kaggle

• All analyzed episodes
• People mentioned on Kaggle
• Topics mentioned on Kaggle