Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Volodymyr Vialactea

What this episode is about

Team Volodymyr Vialactea presents their second place solution for BirdCLEF 2025, detailing their methodology that includes establishing a strong baseline, utilizing transfer learning, and employing advanced techniques like sod labeling, post-processing, and assembling. The presentation also highlights the team's background, challenges faced during validation, and incorporates feedback, insights on failed ideas, and an interactive Q&A session.

Key takeaways

• Great ideas often arise from failed ones; the path to innovation is littered with discarded concepts.
• Cross-validation is a fine art, but beware—noisy data can turn validation metrics into lottery tickets.
• A strong baseline is like a sturdy foundation; without it, your data castle risks becoming a sandcastle.
• Transfer learning: because sometimes you need the wisdom of others instead of reinventing the wheel.
• When sampling and imbalance issues collide, creativity in strategy can turn potential pitfalls into gold mines.

Who speaks in this episode?

• Volodymyr Vialactea (Host) · 21 min speaking
• Fred Nandanda (Guest) · 5 min speaking
• Tom (Guest) · 1 min speaking

What is discussed, and when

1. 0:00 – 5:00: Fred Nandanda, Bird Lab, Better Medicine, Zuk
2. 5:00 – 10:00: Counterto, I Naturalist
3. 10:00 – 15:00: I Naturalist

Notable quotes from "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Volodymyr Vialactea"

• So regarding our background so I'm I'm engaged in several companies. So my main occupation is a computer vision engineer at better medicine. So it's about computer vision aimed…

  about Better Medicine · at 1:00

• We have also incorporated some additional data sets and also from something new that Fernando advised it's label smoothing which also worked pretty well. after that we dive…

  about Fred Nandanda · at 3:20

• So using counterto we have added seven a bit more than 7,000 additional samples and from previous competitions we have added only 90 additional samples. After that we have.

  about Counterto · at 9:07

Who and what is mentioned in this episode?

People

• Fred Nandanda — Person · Referenced collaborator / team member · 2 mentions · first at 2:03
  Introduced as a key contributor to the team’s competition solution, described as a consultant and Kaggle Grandmaster; credited with advising label smoothing.

Companies & orgs

• Better Medicine — Company · Business focus · 1 mention · first at 1:10
  Company where the speaker works as a computer vision engineer, focusing on computer vision in the medical domain.
• Zuk — Company · Business focus · 1 mention · first at 1:20
  Ukrainian defense tech startup co-founded by the speaker, focusing on computer vision with slightly different targets than Better Medicine.
  About computer vision aimed at medical domain. also I'm co-ounder and deep learning team with at zuk. It's Ukrainian defense tech startup which is doing pretty much near same…
• National Technical University of Ukraine — Company · Affiliation · 1 mention · first at 1:33
  University where the speaker is currently working on PhD and giving lectures.
  Doing pretty much near same but with slightly different targets. also I'm currently working on PhD and giving lectures in national technical university of Ukraine eagerosk poly techchnic institute.
• Micros Analytics — Company · Affiliation · 1 mention · first at 1:42
  Ukrainian-founded startup where the speaker spends some time as a machine learning advisor, mostly operating in NLP and cognitive security domain.
  Have also some time with machine learning advisory at Micros Analytics which is also Ukrainian founded startup but mostly operating it in NLP and cognitive security domain. yeah.
• Counterto — Company · Data source · 1 mention · first at 9:07
  Data source used in the competition, adding more than 7,000 additional samples.
• I Naturalist — Company · Data source · 2 mentions · first at 9:25
  Data source tried in the competition, but did not contribute to the score.
  Additional samples and from previous competitions we have added only 90 additional samples. After that we have also tried I natural I naturalist CSA and also we have updated…

Concepts & topics

• Ukrainian Defense Tech — Concept · Industry focus · 1 mention · first at 1:20
  Concept describing the industry focus of Zuk, the Ukrainian defense tech startup.
  About computer vision aimed at medical domain. also I'm co-ounder and deep learning team with at zuk. It's Ukrainian defense tech startup which is doing pretty much near same…
• Bird Lab — Concept · Competition series · 2 mentions · first at 2:34
  Series of competitions where the speaker's experience from previous competitions was used as a baseline.
  Multilabel nature of data. using a set style model and carefully backbones. I think it's like casual approach in bird lab series.

Full transcript with timestamps

0:00 So let's dive into our second place solution. Uh so uh how our presentation is uh is built. So first of all we will uh talk about our background. Then we will summarize uh main points of our solution. After that we will talk about the validation uh which also didn't really work. Uh and then we will go step by step by uh parts of our uh of our solution. So first of all it will be strong baseline. After that it will be transfer learning. Uh next it will be sod labeling uh post-processing and assembling. Uh we will finalize all these points with uh ablation table. Uh then we will add um small section about uh audio creation. Um talk a bit about failed ideas. um g give some our feedback and uh outline resources and finally dive into Q&A.

1:00 Uh so regarding our background so I'm uh I'm engaged in several companies. So uh my main occupation is a computer vision engineer at better medicine. So uh it's about computer vision aimed at medical domain. Uh also I'm co-ounder and deep learning team with at zuk. It's Ukrainian defense tech startup which is doing pretty much uh near uh same but with slightly different uh targets. Uh also I'm currently working on PhD and giving lectures in uh national technical university of Ukraine uh eagerosk poly techchnic institute.

1:42 um have also some uh time with machine learning advisory at Micros Analytics which is also uh Ukrainian uh founded uh startup but mostly operating it in NLP and cognitive security domain. Uh yeah and also Kling a bit. Uh Fred Nandanda is a top consultant uh business and technology consult consultant at uh uh telecom solutions. He's also kgle competition grandmaster. So uh he was kind uh he's pretty kind proficient in both in deep learning and especially in uh classical data science for exploring some stat statistical information and so on.

2:31 Um so regarding our summary in a short uh so we started from strong baseline uh which was uh which was mostly based from our u experience from previous BL competition. uh it was based on augmentations which were used mostly to address domain shift. Uh data sampling strategy was also very important because of uh high class imbalance and uh macro uh nature of our metric. Uh use of secondary labels again to address multilabel uh nature of uh of data. Uh using a set style model and carefully backbones. I think it's like uh casual approach in in bird lab series.

3:20 Uh we have also incorporated some additional data sets uh and also from uh something new that Fernando advised it's label smoothing which also worked pretty well. Uh after that we dive deeper in some more advanced uh approaches in order to enhance enhance our solution. uh it was in domain transfer learning uh pseudo labeling post-processing and and assembling. Uh so talking a bit our validation it uh was pretty straightforward. So we use classical five holes uh cross validation. We stratified by primary label. Uh and we also grouped by author in order not in order at least to decrease leakage between possible uh same recordings that were made by the same author maybe in nearly at the same time let's say. uh then we uh we had problems that sometimes uh species may uh may um uh may get only into one fold.

4:25 So uh there are two ways how we can deal with it. So first way is uh that we can uh put at least one sample of each species at each validation fault in in such a way we will be able to compute uh like all uh all classwise rocks for each species. Um maybe having a better representative metric. uh but at the same time we need to take into account that uh matrix for uh species which have for example only one sample or or two samples they will be very noisy. Uh and another approach is when we take all under sample species and exclude them from validation faults and put them in training faults. Uh in this way we can increase the presence of um uh of these under sample species uh in our training set. Uh which uh should uh in theory increase our model performance on these under sample species. Uh but at the same time we we do not understand our validation scores on them. uh but also regarding validation influence it might

5:34 even increase our validation overall like over all on the whole set because we we will not have this very noisy species in the validation faults. Uh if you take a look at the at the validation local validation course compared with uh uh public leaderboard you can see that the correlation is is really bad. uh but if we take for example zone up to 0.9 uh public leaderboard there is still some kind of uh of correlation. Uh but when we cross this line uh like it completely mess up. Um so regarding our baseline uh what what we have added into it. So first of all it is augmentations. We have used mostly four four main ones. Uh it is classical mixup. Uh we have also mixed in audio form domain and we um and we and we take uh element wise max across across two samples labels.

6:40 Uh we have also mixed some backgrounds. So we have taken um envir environmental soundsscapes. Um I I might be a bit u messing up with with the name of this data set but it's kind of like well-known data set. Uh and also we uh we had uh we have downloaded the previous soundscapes and take only like no call uh no call regions. Uh also we have used spec uh spec which just kind of cut out across frequencies and time frames. uh random filtering is kind of random equalizer. Um yeah so it's like the our main for uh augmentations.

7:27 Uh then regard uh regarding data sampling strategy I think it's pretty important for us. Uh so there are two approaches that we have used. First approach is just to use balance sampling across all classes. uh if if we look at this formula it's like this um this approach when when the gamma equals to minus one another approach when the gamma is uh equals to minus 0.5 it's uh a bit smoothing uh smoothing sampling so uh more frequent classes will appear a bit more frequently than in balanced one uh but in order to again upsample undersled under sampled classes we just repeated them in our training data set up to 100. So we take all um all species that that have less than 100 species and upsample them up to 100. Uh and to use it with is a combination of this uh data sampling strategy. Uh regarding backbones uh we have used NFnet uh L uh L0 and efficient net v2 small uh and label smoothing was pretty much

8:40 in a classical way uh with al alpha 0.05 05. Um so here is um yeah and also regarding additional data that we have used. So we um uh we have tried several data sources. So uh eventually we have started from classically we have started from accountant and from previous competitions. Uh so using uh counterto we have uh added seven uh a bit more than 7,000 additional samples and from previous competitions we have added only uh 90 uh additional samples. After that we have also tried I natural I naturalist CSA and also we have um updated our sento dump uh with latest samples but unfortunately adding these uh three more sources uh did not contribute to our score and using uh this uh these two additional sources was like um our solution.

9:50 uh if we're talking about like ablation on this additional data. So um mostly we can see that uh according to public score even using the only training only train data uh appears to to work best. Uh but when we but when we add uh like our two selected additional uh uh data sets uh it uh it works better on private. So yeah uh we didn't know it but uh on our intuition it it should improve. So we decided to use this uh second approach. Uh also a small oblation regarding um augmentation. It also uh it also appears to us a bit counterintuitive uh because uh as you can see uh as you can see for example uh not using mixup uh using and not using mixup by some reason uh perform nearly the same on on our enhanced baseline setup.

10:58 um uh on a public score uh and it it worked much better on the private score. Uh using uh using another augmentation also uh gave some uh gave some improvement on our public score and also gave improvement on private score. I think the most interesting part here that by some reason uh when when we didn't use um mix up um like the the public score did not change. Uh yeah we have made like all this ablation after the competition ends. Uh so yeah it was just an interesting finding for us.

11:39 Yeah Tom uh small question. What was a what is CSA? I don't know that I might not be aware of that one.

11:46 Oh, it's um it's like one of the data sources uh that was also used in uh bird uh like I naturalist and the second one was CSA. I think it's like um cannot remember like the name of the university where it was taken.

12:03 A lot of the Colombia recordings uh were in CSA in the CSA data set. Okay. Okay. with, you know, there were like people speaking Spanish in the background and so on. A lot of them were from CSA.

12:18 Uh, yeah. Cool. Sorry. Yeah, I Yeah, I remember that one. Yeah. Thank you. Yeah. So, let's go further to transfer learning one. So, how uh how did we do it? So the idea is just that we have lots of uh like very big dump of counter with lots of species but we cannot really use it for our um current uh species. Uh so why not just to download it and use as kind of pre-train in order to finetune after that on our uh species of interest. So we have collected uh 16,000 issues from uh all birdcl tax tonomies from previous years from this year. Uh then we downloaded all recordings and metadata meta data from Santa. Uh we have filtered it. So we remove species uh from bird from this year BCL in order to avoid data leakage.

13:19 remove corrupted files, invalid uh invalid files, uh also files that do not comply with licensing. Um also we have removed species which uh have less than 10 recordings in order not to have like this u sample species and maybe some training uh training instability. And we uh and we come up with a uh pre-training data set which um which had uh 890K recordings and 700 7 and half species. Uh after that we split it into five person for hold out and other for training and uh while validation we validated only on species which had more than 100 samples.

14:10 uh we trained without class balancing but using like similarly the same setup as uh in our enhanced baseline logic uh for checkpoint we have taken only uh only our pre-trained uh CNN bed bone uh and we have selected uh the best checkpoint by micro rock metric uh and after after selecting this checkpoint pertain checkpoint we just used this backbone as a starting point uh and fine-tuned uh as uh as we described previously in our uh enhanced baseline without any changes to learning rate or scheduleuling strategy. Um yeah, it it helped us uh it uh it reasonably helped to improve our score but uh next we decided to move further and um the next step was pseudo labeling. Uh so overall scheme was uh next one. So we have taken uh our uh trained models uh predicted on soundsscapes uh then take uh then take probabilities.

15:23 So we do not uh we do not hard label them. We um we store probabilities for each 5-second chunk. Uh after that we uh applied uh we selected uh uh some particular soundsscapes. We will uh we will elaborate uh on it on the second slide. Uh and then just add it to our u training set uh and applied uh some specific sampling strategy in order to combine our label data set and our uh initial training set. Um after that we trained model uh obtained a new set of models trained with with a um with this set of pseudo labels and repeat to and iterate to the next iteration. So for the next iteration what what we did we removed uh all soundscapes that we are predict on the previous iteration uh and also added uh uh and after that added them uh into our training set. So we we do not like predict on on the same labels that were used in in in the training set because it's like an obvious leakage.

16:40 Um another approach uh in order to uh in in in in order to fix our kind of um yeah in order to fix like previous predictions. So what I have mentioned previously when we do not predict on uh previously predicted soundsscapes uh it can be a problem because when we improve our models with sud labeling um it makes sense to use like uh new models in order to predict all s labels again. So uh these uh uh models from the previous situations much uh should be like be much powerful than models without any labels. Uh and it can be also done but in order to do it we need to split our soundsscapes in uh several faults uh and then apply uh out of fold prediction strategy.

17:41 In such a case we will we will not have kind of uh leakage uh a kind of leakage. So if we just split our se soundscapes in several folds then predict them uh then concatenate then then use each soundsscape fold for each trained data fold and mix uh only on the fold wise manner. Uh train models for each fold and for for each architecture. uh then we can use uh and rep predict all soundscapes on each iteration. Um finally regarding our uh sampling strategy so uh let's assume we have already um have our pseudo labeled uh data set and we have our train data set.

18:29 So uh on each sampling step we need to decide uh which sample do uh do we take uh and in order to do it we introduce uh this scheme. It's not it's not complex. So the idea that firstly we sample our train sample which has uh bird class maybe several classes uh and audio. uh then we check if the bird class like the pime the primary class is present in our p s p s p s p s p s p s p s p s p s p s pseudo data set uh if it is present we will take sudo sample with a pro uh with some probability uh in our case it's like 0.4 four. Uh in this case we will use tilda sample. If the class is not present or we uh we are not lucky with sampling uh with sampling this probability we just use our initial train sample.

19:26 Um yeah and um regarding our approach how we how we overall predict and select our pseudo labels. So uh we uh for for each iteration we train mostly five-fold nfnet and efficient net. So overall we had 10 models for each iteration in order to predict uh the next iteration of pseudo labels. And when we have predicted all pseudo labels uh with soft labels we uh used next logic in order to pick the the chosen one. So for each 5-second chunk we computed the maximum predicted plus probability. Uh after that we have thrown away all chunks that have this maximum probability uh less than 0.5.

20:14 Uh and then we uh zero out all probabilities for selected chunks that are less than 0.1. Uh and after that while training as we have used binary cross entropy we have used a mix of binary cross entropy and focal loss uh we were able to train uh using uh soft targets. Uh so we do we do not hard labelled our obser labels. We use uh direct soft targets and the idea was that uh it it was not only kind of serial labeling also we uh we have seen it as kind model distillation because we have average like 10 models and we trained on this on the distribution of 10 models we have trained like only one model let's say uh and also like zeroing out these uh low probabilities is kind of removing noise because If we assume that if the model uh gives this low probabilities mostly it will be kind of very noisy distribution uh for these uh low probability classes. Uh when we mix up uh uh also while training we could mix

21:26 up both train and two label samples. We could mix up train and train and to the label and to the label. uh in such a way we we could mix up like our hard labels with uh soft targets. Uh so it it was also possible and in order to make it like correctly we just sum up and click to 0.1 uh sorry and clip from zero to one. Uh we have trained like several options. So uh uh we have shown like two methodologies when we predict uh C labels in autofall autofold mode uh and when we predict like just full S label data set. Uh so we have trained using both options and uh in the final final ensemble we have like models from uh from trained in different methodologies.

22:19 Uh in the table we can see the number of selected chunks for each iteration. So on the very first iteration we obviously get the uh maximum number of selected files. So uh it's like uh four uh four and nearly a half thousand soundscapes and uh nearly 20,000 5-second chunks. Uh after that we have a bit less number of files not a bit but much less number of files and chunks. And on the third iteration, we also had uh nearly the same number of files but a bit more number of selected chunks. Um yeah, regarding our post-processing, it really contributed uh reasonable score. So we have tried two approaches.

23:13 Okay. Uh we have tried two approaches. It was like a mean uh when we just average probability across uh across the whole soundsscape and multiply it for each chunk. And we have tried taking on only the probabilities of top uh of top most probable values and the second one worked best when we when we put it like top and top top class probabilities to to one. Uh regarding ensembling um we have tried like two approaches. one is very naive when we just select best public one. Uh and we have selected it for the final solution and the uh and the second approach was to use of tuna in order to maximize um our validation score out of fault validation score. Uh it was a bit worse on uh on public but uh it might result in in a bit better score on uh on private.

24:12 uh regarding ablation here is a table. So we can see that uh we get a pretty reasonable gain from uh transfer learning. Uh the next uh the next important milestone was using p labels. We can see that uh we situate on a on a on on a search uh on on a search um on a search iteration uh and using postprocessing also contributed reasonably to our score especially on private video bot uh coration. So yeah I will just go pretty fast. So regarding creation there there is several artifacts for example TTS speech uh like uh casual human voice and some long uh long silent chunks.

25:04 Uh so we have tried several ways applying what uh manually expecting samples applying model probabilities in autofold mode uh and also applying um only these techniques for under sample species. uh mostly um uh they did not show some consistent results but overall manual correation on under sampled species worked uh but didn't give some big big advantage. So we have used just sampling from five second random chunk from the whole audio uh from the failed ideas. So transfer learning on bigger data sets labeling with more iterations or chunk wise with train audio. Uh we have also tried some other uh ways of post-processing.

25:55 Uh interesting is L2 uh L2 model. Uh I think it should work but maybe it will require some more efforts to to invest into it. Uh so yeah I think the main feedback is just that maybe soundscapes for validation. Uh regarding all other points I think everything was really nice. um some resources. Uh so thanks to armed force greencliff and lifeclft team and also to cargo team who has hosted this competition.

Questions about this episode

Who speaks in "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Volodymyr Vialactea"?

Volodymyr Vialactea (Host), Fred Nandanda (Guest), Tom (Guest).

What is "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Volodymyr Vialactea" about?

Team Volodymyr Vialactea presents their second place solution for BirdCLEF 2025, detailing their methodology that includes establishing a strong baseline, utilizing transfer learning, and employing advanced techniques like sod labeling, post-processing, and assembling. The presentation also highlights the team's background, challenges faced during validation, and incorporates feedback, insights…

What is mentioned in this Kaggle episode?

9 people, works and ideas are mentioned, including Better Medicine, Zuk, Fred Nandanda, Ukrainian Defense Tech, National Technical University of Ukraine, Bird Lab. Each one on this page links to its own page with every episode that mentions it.

How long is "Kaggle Winners Walkthroughs: BirdCLEF 2025 with Team Volodymyr Vialactea"?

The episode runs 27 min; the transcript is about 3,630 words in 30 timestamped sections.

More from Kaggle

• All analyzed episodes
• People mentioned on Kaggle
• Topics mentioned on Kaggle