Kaggle Winners Walkthroughs: Child Mind Institute —Problematic Internet Use with Team Lennart Haupts

What this episode is about

The presentation outlines the solution developed by Nar House for a Kaggle competition on problematic internet use, emphasizing his background in economics and machine learning. Key points include his strategy of predicting underlying scores for better accuracy, using an ensemble of models like LightGBM and XGBoost, as well as highlighting significant features such as age, daily internet use, and sleep disturbance scores.

Key takeaways

• Winning isn't just about predicting; it's about optimizing your strategy—like predicting scores over labels.
• In machine learning, the noise is sometimes louder than the signal; PCA to the rescue, but with caution!
• Feature selection is more art than science; eyeballing and manual selection can yield gold alongside the metrics.
• Overfitting to leaderboards? Always a risk. Stepping back can foster resilience and better results in competitions.
• Sometimes your model's choices reveal surprising truths—who knew sleep disturbances would be such a pivotal feature?

Who speaks in this episode?

• Nar House (Host) · 11 min speaking

What is discussed, and when

1. 0:00 – 3:00: Nar House, XG Boost, Cat Boost, Kegle
2. 3:00 – 6:00: Nar House, PCA
3. 6:00 – 9:00: XG Boost, Cat Boost, LightGBM, Optuna
4. 9:00 – 12:00: XG Boost

Notable quotes from "Kaggle Winners Walkthroughs: Child Mind Institute —Problematic Internet Use with Team Lennart Haupts"

• I'm Nar House and I'm here to share my solution to the competition. to the agenda, first I'll give you some background information on myself. Then I will.

  about Nar House · at 0:04

• Scores and converted them to the labels via optimized thresholds. And the final model was then a majority voted ensemble of treebased models one lightm model, one cat boost model, two xg boost regressors and one extra trees.

  about XG Boost · at 1:39

• Only 15 components that captured about 82% of the variance. A deliberately cautious approach. While the overall impact of those features was relatively small, there might be still some valuable information in the raw features.

  about PCA · at 3:15

• Optuna combined with repeated stratified k-fold cross validation to ensure robust results. while this was computationally very expensive, this approach helped mitigate the optimizer course which was a key challenge in this competition.

  about Optuna · at 8:38

Who and what is mentioned in this episode?

People

• Nar House — Person · Main speaker · 2 mentions · first at 0:04
  Presents and explains their winning competition solution, including feature engineering, model ensembling, and validation strategy.

Companies & orgs

• Kegle — Company · Platform for competitions · 1 mention · first at 1:17
  The speaker participated in Kegle competitions, which is relevant to their experience in machine learning
  Early on I noticed that overfitting to the leaderboard may become a problem in this competition as the submissions were very sensitive to the selected seat. So I stepped back from.
• Higho University — Company · Affiliation of speaker · 1 mention · first at 0:33
  The speaker is an economics student at Higho University, which provides context for their background
  Then I'll share some important findings and simplified model first about my background I'm an economics student at higho university dissor with a strong focus on econometrics machine learning and empirical.

Products & tech

• XG Boost — Product · Key component in ensemble model · 3 mentions · first at 1:58
  Used as part of the majority voted ensemble of tree-based models in the competition solution
• Cat Boost — Product · Key component in ensemble model · 2 mentions · first at 1:48
  Used as part of the majority voted ensemble of tree-based models in the competition solution
  Scores and converted them to the labels via optimized thresholds. And the final model was then a majority voted ensemble of treebased models one lightm model, one cat boost model, two xg boost regressors and one extra trees.
• LightGBM — Product · Key component in ensemble model · 1 mention · first at 8:24
  Used as part of the majority voted ensemble of tree-based models in the competition solution
  Scores and converted them to the labels via optimized thresholds. And the final model was then a majority voted ensemble of treebased models one lightm model, one cat boost model, two xg boost regressors and one extra trees.
• Optuna — Product · Hyperparameter optimization tool · 1 mention · first at 8:36
  Used to optimize hyperparameters in the training methods

Concepts & topics

• PCA — Concept · Dimensionality reduction technique · 1 mention · first at 3:12
  Used to reduce dimensionality in the feature selection and engineering process

Full transcript with timestamps

0:00 Um, first of all, thank you to anyone who made this uh competition possible. I'm Nar House and I'm here to share my solution to the competition. Um, to the agenda, first I'll give you some background information on myself. Then I will summarize my solution. Then I'll dive deeper into the feature selection and feature engineering and the training methods. um then I'll share some important findings and um simplified model first about my background um I'm an economics student at higho university dissor with a strong focus on econometrics machine learning and empirical methods over the past four years I've participated in kegle competitions on and off combining them with my own machine learning and analytics projects.

0:57 Uh early on I noticed that overfitting to the leaderboard may become a problem in this uh competition as the submissions were very sensitive to the selected seat. So I stepped back from the competition after a while as I was afraid of overfitting to the leaderboard and I joined at the end of the competition again to make my solution a bit more robust. But uh never did I think I would actually win. So now to summarize my uh solution, instead of uh predicting the target directly, I um predicted the underlying scores and uh converted them to the labels um via optimized thresholds. And the final model was then a majority voted ensemble of treebased models um one um lightm model, one cat boost model, two xg boost regressors and one extra trees regressor. Um the key features were largely not surprising. Uh age, weight, height, the daily internet use and um another very important feature was the

2:13 score of the sleep disturbance scale. to give some uh to summarize the runtime of my solution. Pre-processing takes about 2 minutes. Uh training including cross validation about 3 minutes and uh parameter tuning took um a bit longer a few hours due to the repeated stratified kfold uh cross validation approach. Um yeah now to go into more detail on the feature selection and engineering. Um first um let me u tell you how I approached the acigraph data. Um I created separate masks for day and night deriving some uh descriptive statistics.

3:04 Uh here the number of features grew very quickly with high colinearity and very little signal. So to re reduce dimensionality, I applied PCA retaining only 15 components that captured about 82% of the variance. A deliberately cautious approach. While the overall impact of those features was relatively small, there might be still some valuable information in the raw features. Um I also um addressed dimensionality due through uh reducing the dimensions of the fitness gram zones by um combining them to mean minimum and maximum features and dropping the original columns. I also created some derived features such as the difference between daily energy expenditure expenditure and the basil metabolic rate and the ratio of extracellular to intracellular water. Especially the first one of those I had moderate uh showed a moderate improvement of my CB score and to deal with uh noise towards the end of the

4:17 competition I quantile wind 15 features. To be honest, my approach here wasn't particularly um methodical. I just u selected a few uh features and uh looked how the score and changed. This uh contributed to an improvement in my CV score and uh to the stability on the public leaderboard. And um for feature selection I performed manual backwards selection um by looking at the feature importance and uh generally eyeballing uh the changes in the CV score. features were deemed not important when they were jointly unimportant in uh three of the models and unimportant is uh means they were among the u bottom 10 features.

5:17 Um considering this was an ensemble uh model uh pointing towards the most important features isn't that straightforward for me. Uh but among the consistently important features um there were age and the total raw score of the sleep disturbance scale and the t-core of the sleep disturbance scale as well as the height, the weight and the hours of uh internet use. Here are now three of the feature importance plots um of the five models. It's still not um that clean of a a graphic, but I guess one can still see um that what I just said that the sleep disturbance scale is very important, age is very important and um there were other important features like especially from the fitness gram um uh features here the normalized pull-ups or curl ups, but it was relatively inconsistent which of the these features were prioritized by the model. Um, moving on to the um partial dependence plots, I want to

6:38 show how the average prediction changes with changes in the features. Um, and yeah, those are the feature importance plots for three of the most important features uh for weight. The prediction sharply increased between 40 and 80. As you can see, plateaued and then there was around 120 pounds. Again, a sharp increase. Um, for the sleep disturbance scale, we see a relatively uh steady increase with um somewhat of a jump between 40 and 45. Um and then for the internet hours which was originally uh just an integer feature but due to uh last imputation there were also some floats in it. You can see that uh between zero and one there's a steep upward curve followed by a small plateau and another upward curve. Something quite interesting here is that the signal of weight is largely due to um age and sex. And um uh after partialing out, one can see that the uh signal diminishes um rapidly but um there still remains some signal

8:11 for the score of the sleep distortion scale. If one partials out age and sex, there still remains a strong signal. Uh now moving on to the training methods. As I already said, the um final model was a majority voted ensemble of large GVM cat boost extra trees and two extra boost regressors. The hyperparameters were optimized using optuna uh combined with repeated stratified k-fold cross validation to ensure robust results. Uh while this was computationally very expensive, this approach helped mitigate the optimizer course which was a key challenge in this competition.

8:57 Um the models in this ensemble were correlated between 0.95 and.99 with the um um initially the two extra boost regressors had different parameters and thus lower correlation but over time they converged and now they are essentially basically the same model just with double the weight and ensemble. Um well and that's it for the training method. Moving on to some important findings. Um throughout this competition several key fe insights emerged. Quantile binning and feature selection improved the performance as well as the stability.

9:49 Um a robust t-fold stratified kfold cross validation approach stratified by bins was essential for the reliable validation. Uh changing seats frequently allowed me to test the model's robustness both in the CV and on the leaderboard. On the leaderboard I largely only looked at the variance of my scores. I submitted multiple uh times the same solution just with different seats to see how it changes. Um lastly uh robust lastly while robustness was key to my approach I'd like to acknowledge that luck was a very important factor in my uh success. U moving on to simplified model. Um for the simplified model I selected 18 features um dropping all PC all act features and uh only using a single XG boost regressor. Um, one of the key features of the XG boost regressor could be um is that I used parallel trees essentially having a boosted forest. Um, the public leaderboard suffered the most under some

11:12 uh simplification dropping to 425. the private leaderboard score dropped to 476 and the optimized CV score to 453. Um now to summarize the u solution um early on I recognized that overfitting would be a significant challenge. So emphasized strong cross validation strategies and reduced model complexity. Feature engineering and uh elimination play critical roles in enhancing the model's performance. And while this was um instrumental in the good performance that was a key aspect of my success was luck overall. This is a very traditional approach and I hope I could still provide some value. Um, and now I'd be open for questions.

Questions about this episode

Who speaks in "Kaggle Winners Walkthroughs: Child Mind Institute —Problematic Internet Use with Team Lennart Haupts"?

Nar House (Host).

What is "Kaggle Winners Walkthroughs: Child Mind Institute —Problematic Internet Use with Team Lennart Haupts" about?

The presentation outlines the solution developed by Nar House for a Kaggle competition on problematic internet use, emphasizing his background in economics and machine learning. Key points include his strategy of predicting underlying scores for better accuracy, using an ensemble of models like LightGBM and XGBoost, as well as highlighting significant features such as age, daily internet use,…

What is mentioned in this Kaggle episode?

8 people, works and ideas are mentioned, including Nar House, XG Boost, Cat Boost, LightGBM, Optuna, Kegle. Each one on this page links to its own page with every episode that mentions it.

How long is "Kaggle Winners Walkthroughs: Child Mind Institute —Problematic Internet Use with Team Lennart Haupts"?

The episode runs 12 min; the transcript is about 1,406 words in 11 timestamped sections.

More from Kaggle

• All analyzed episodes
• People mentioned on Kaggle
• Topics mentioned on Kaggle