Kaggle Winners Walkthroughs: Child Mind Institute — Problematic Internet Use with Team sqrt4kaido

What this episode is about

The presentation provides an overview of the author's experience in a Kaggle competition focused on problematic internet use, highlighting their background in information science and machine learning, as well as their journey to becoming a Kaggle Grandmaster. Key points include the core modeling techniques utilized in the project, the processing steps for feature engineering and optimization, and the variation strategies implemented for robust model evaluation.

Key takeaways

• Deep learning isn't always the holy grail; sometimes, light GBM steals the show with its robustness.
• Sudding isn't just a typo; it's a game-changing strategy that boosted my score significantly!
• Imputation and thresholds: the unsung heroes of data preprocessing that can make or break your model.
• In a world of complex models, simplicity in feature engineering often delivers the best performance.
• Kaggle grandmaster status: proof that sometimes, the road less traveled (like tree models) leads to victory.

Who speaks in this episode?

• Team sqrt4kaido (Host) · 17 min speaking

What is discussed, and when

1. 0:00 – 5:00: Kag
2. 5:00 – 10:00: S levels, pandas
3. 10:00 – 15:00: percentiles
4. 15:00 – 20:00: percentiles

Notable quotes from "Kaggle Winners Walkthroughs: Child Mind Institute — Problematic Internet Use with Team sqrt4kaido"

• Data as well as for tabular data. These skills were highly beneficial for this competition. Moreover, as a result of the for this competition, I was able to achieve the title of Kag competition grandmaster.

  about Kag · at 1:55

• Solution and significantly boosted the private leader score. Here's how I created and used S levels. First, I performed fivefold time tension cross valition using the loads where SI score was not N to train models.

  about S levels · at 5:03

• Metrics listed in statistics codes and use them as features. The reference notebook initially used pandas for this process but I reload it using pulas. sign significantly reducing the comput computation.

  about pandas · at 8:50

• Specifically the initial values were deliver derived as the average predict value for each grant level. For more details please refer to this car.

  about percentiles · at 11:18

• Next, let me explain the ensemble strategy. For the final submission, I utilized voting result from a total of 200 models. Specifically, these included regression and classification models built with both light JVM and XG boost.

  about LightGBM · at 13:00

Who and what is mentioned in this episode?

Companies & orgs

• Kag — Company · Competition platform · 1 mention · first at 1:39
  Platform where the competition was hosted, and where the presenter achieved the title of grandmaster.

Products & tech

• LightGBM — Product · Core modeling technique · 3 mentions
  Primary machine learning model used in the competition solution, achieving the best performance.
• XGBoost — Product · Secondary modeling technique · 3 mentions
  Secondary machine learning model used in the competition solution, achieving the second best performance.
  Each of these four types of models was trained with a fivefold crossition repeated over 10 different sheets. Okay. Lastly, I'll l up with the summary of my approach.
• pandas — Product · Data processing tool · 2 mentions · first at 9:00
  Initial tool used for statistical calculations, later replaced by Polars for better performance.

Concepts & topics

• percentiles — Concept · Optimization strategy · 5 mentions · first at 10:19
  Key method used for threshold optimization, providing greater stability and accuracy in predictions.
• S levels — Concept · Data augmentation technique · 2 mentions · first at 5:08
  Pseudo-labeling method that significantly improved the private leaderboard score by augmenting the training data.

Full transcript with timestamps

0:00 I stop my presentation. First of all, I would like to express my sincere thanks for organizing this competition. It was it was an incredibly challenging experience, but I learned so much especially in terms of robust evolution strategies. Thank you again for this valuable opportunity. Next is the agenda slide. Here is the agenda for today's presentation. First, I'd like to start by introducing myself briefly. Then I will walk you through the summary of my approach. After that, I will explain the specific method used in my solution.

0:49 Finally, I'll wrap up by summarizing the overall solution once again. Okay, let's get started. Let me start with a brief introduction about myself. I majored in information and image science at university when I conducted research primarily on image processing. After graduation, I began my career and I embedded engineer. Later I transitioned to a startup specializing in anomaly detection and aostic processing. Aostic proing which is where I first encountered machine learning and deep learning. Currently I work at a company specializing in AI contract developing project and image processing.

1:39 I'm also active participant on cargo. Throughout my career, I have gained expens extensive experience in preprocessing and building models for sequential data such as audio and sensor data as well as for tabular data. These skills were highly beneficial for this competition. Moreover, as a result of the for this competition, I was able to achieve the title of Kag competition grandmaster. Okay, up next is the this uh my solution summary slide. Here is a summary of the model. I'll explain it in three main categories. First, I'll discuss the core modeling techniques such as S leveling models and loss functions.

2:39 Next, I'll explain the processing steps including future engineering and solution optimization. And finally, I'll cover the variation strategy used to robustly evalate and evate the model. Okay, first key modeling techniques. Let's talk about the model for this competition. And I primarily focused on treebased models. While I experiment while I experimented with neural networks, they didn't perform as well and the results varied significantly across folds and therefore I decided not to pursue them further. The best performing models of my solution was based on light GBM for the ensemble. I incorporated incorporated XD boost which had the second best performance.

3:44 I consider several types of loss functions but 3D loss deliver the best accuracy in my case while I wouldn't describe the target distribution as long tail but I believe three loss fit well because excluding num values the distribution had a peak at zero and decreased as integers increased although I also used used multrass classification in the ensemble. It contributed very little to the overall score improvement and I initially used the QWK optimization loss interested in Jackson's public notebook but I decided not to adopt it because it required a long training time.

4:42 Similarly, I did not use LMS loss as I determined that TW loss was sufficient for regression tasks. Next, I would like to explain Suding which was a key element in my solution and significantly boosted the private leader score. Here's how I created and used S levels. First, I performed fivefold time tension cross valition using the loads where SI score was not N to train models. Then I used these trend models to make predictions on lows where SI was now applying voting across 50 models to generate it generate S levels. And next I combined the generate s levels with the loads where SI was not and retrained the model to avoid overfitting to the S levels. I split the S levels load into five new F dying this stage which was another consider another key consideration. As a result of these measures, my private leaderable score improved significantly from 0.48 to 0.468. Initially, I didn't expect suding

6:23 to have such a strong impact based on the cross validation and leaderboard scores. However, considering that other solution seems to benefit from increasing the number of folds, it's likely that augmenting the training dead size was effective. Then now let's move on to the next section. Important preprocessing and features features. This slide outlines how I ensure lowest validation by focusing on two key processes. Imputation of non values and threshold determination. For non value imputation, I use the varian of each gram from the training set for both validation and test data. While num values are generally less problematic problem problematic for decision tree models but I found it crucial to carefully test imputation method in case like this competition where over half of the sensor data ID contains n values. Regarding threshold, threshold determination thresholds are calculate using calculated using the training set

7:48 and validated on each fold variation set. The detailed method for determining threshold will be explained later. Okay. In this slide I I will explain the future features used from tabular data and sensor data. First regarding the feature of the tabular data, I utilized all all features in the data set. In particular, categorical in integer values were treated not only as categorical data but also as numerical data which improved the overall score. Next about the features of the sensor data for the crown indicated in use codes. I calculated the stat statistical metrics listed in statistics codes and use them as features. The reference notebook initially used pandas for this process but I reload it using pulas. sign significantly reducing the comput computation time. Interestingly, Boris stat statistical calculation functions include ner count which is not part of the equivalent pandas functions. By including this new count

9:30 as a feature, I observed a slight improvement in the score. This may reflect a pattern where severely ill patient tend to abandon sensor measurement midway through the process. As a side note, I also spent time trying to build an AS models to directly learn the time series feature of the sensor data. However, I eventually gave up on this approach as it failed to achieve satisfactory scores. Okay, let's now focus on threshold optimization. I experience optimization using pass percentiles which played a key role in improving improving both the stability and accuracy of my solutions predictions.

10:31 First about about the optimization process percentile based threshold optimization provides greater stability compared to direct numerical optimization. This method was introduced as a solution in a previous competition that adopted diagram score and I thought that would also be effective for the metric in this competition. Next about initialization. Initialization the initial threshold were not set as con constants but were instead calculated based on the predictions from the training data. Specifically the initial values were deliver derived as the average predict value for each grant level. For more details please refer to this car discussion. As previously mentioned my social optimization is performed using percentiles. So these initial values are then converted into percentiles.

11:49 Finally, let me explain how the model was evaluated. First, I I'd like to explain the setup for cross validation to train and validate the model. I used a five-fold certified carefold approach. However, significant variability in scores was observed depending on the random seed. So to address this, I evaluated the combined results of 10 seed ving to ensure robustness. Additionally, for each sheet, I optimized the model parameters using optional. When evaluating the result of my experiment, I didn't focus on only single score. Instead, I looked at both the voting scores from all five five-fold across 10 seats and the average scores from the 10 sheets. For example, the best score achieved by single model is as follows.

13:00 Next, let me explain the explain the ensemble strategy. For the final submission, I utilized voting result from a total of 200 models. Specifically, these included regression and classification models built with both light JVM and XG boost. Each of these four types of models was trained with a fivefold crossition repeated over 10 different sheets. Okay. Lastly, I'll l up with the summary of my approach. Looking at the top solutions from this competition, I believe that robust evaluation and data augmentation for the training set and nonvariant amputation were particularly important. My solution in particular stood out stood out in terms of logistic variation and data augmentation compared to other participants.

14:12 All the point listed here are crucial but my personal favorites are the two highlighted in bold. Okay, this concludes my solution presentation. Thank you for your attention. If you have any questions, I'll happy to answer them. Thank you. Yeah, thank you very much. It was a great presentation. Um I actually related to one of the things that's bolded on the screen. I just wanted to hear if you could elaborate a little bit more on the threshold optimization using percentiles both just a little bit more precisely what is it it exactly that you you mean like how did what what is it that you're you're talking about here and then how did you kind of land on this approach as opposed to you know just classic optimization on the score targets.

15:04 Okay, just a minute. Okay. The first uh social optimization by percentile it's uh sorry the in case it's the training set distribution and test test set distribution is similar case. In similar case, the threshold optimization by passenger is very robust in my experience. So like

16:35 equin threshold. I I tried to this appro I tried this approach at least once and honestly I confirmed the approach is effective effective I adopt this approach So honestly uh honestly I tried at at least once and I experience many times I experienced many times that approach is effective. Okay. Yeah. So, just to make sure I understand, by using percentiles instead of the raw scores, it is more robust when the training and testing distributions are imbalanced. Is that what you're saying? Yes. Yes. Okay. And you can do you compute the percentiles within each fold or how do you are you computing them over the entire data set or or how do you compute the percentiles?

18:03 Okay. In my approach, uh thresholds for calculated using the train set and validate uh thresholds were calculated. using the training set and validated on each fold validation sets. So I calculated the threshold per fold per fold and for example I made 200 models I calculated 200 thresholds. I see. Okay. So part of the training process is learning the percentile thresholds that your model is predicting. Um okay.

19:28 Interesting. Does anyone else have other questions? Sounds like the pseudo labeling gave you your biggest boost in accuracy performance. Was was there anything on the sensor processing on the feature engineering or pre-processing that you noticed was you know a significant jump when you were working through that data? Oh, honestly I didn't find the jump up point from sensor data. Uh first I tried to build an neural network models from the sensor data. But it is very difficult because the uh the patient who does not have sensor data.

20:47 Uh the number of patient who has who doesn't have sensor data is very yeah now okay sorry so I gave up this approach and the public notebook introduced this stat statistics approach. So I use that notebook but reload by Borus. Okay.

Questions about this episode

Who speaks in "Kaggle Winners Walkthroughs: Child Mind Institute — Problematic Internet Use with Team sqrt4kaido"?

Team sqrt4kaido (Host).

What is "Kaggle Winners Walkthroughs: Child Mind Institute — Problematic Internet Use with Team sqrt4kaido" about?

The presentation provides an overview of the author's experience in a Kaggle competition focused on problematic internet use, highlighting their background in information science and machine learning, as well as their journey to becoming a Kaggle Grandmaster. Key points include the core modeling techniques utilized in the project, the processing steps for feature engineering and optimization,…

What is mentioned in this Kaggle episode?

6 people, works and ideas are mentioned, including LightGBM, percentiles, S levels, XGBoost, Kag, pandas. Each one on this page links to its own page with every episode that mentions it.

How long is "Kaggle Winners Walkthroughs: Child Mind Institute — Problematic Internet Use with Team sqrt4kaido"?

The episode runs 21 min; the transcript is about 1,897 words in 18 timestamped sections.

More from Kaggle

• All analyzed episodes
• People mentioned on Kaggle
• Topics mentioned on Kaggle