Kaggle Solution Walkthroughs: UM - Game-Playing Strength of MCTS Variants with Team James Day

What this episode is about

The content presents a first-place solution to a machine learning competition focused on predicting the game-playing strength of Monte Carlo Tree Search (MCTS) variants. The discussion covers the author's background, methodology for feature selection and engineering, as well as the training techniques and key findings behind their successful predictive model, which combines various machine learning algorithms to enhance prediction accuracy.

Key takeaways

• Feature engineering is the unsung hero of machine learning; even the tiniest tweaks can alter the game.
• The right balance metric can be worth its weight in gold; skip the noise, seek clarity through tree search.
• Stacked ensembles: because why settle for one model's opinion when you can harness the wisdom of many?
• Correlation is not causation; remember, a linear trend can be misleading in the game of strategy optimization.
• In the data wrangling battle, a pencil and paper are often more valuable than the latest AI toolkit.

Who speaks in this episode?

• James Day (Host) · 11 min speaking

What is discussed, and when

1. 0:00 – 3:00: CatBoost, Tre Search, Kaggle, LightGBM
2. 3:00 – 6:00: Tre Search, CatBoost
3. 6:00 – 9:00: Isotonic Regression
4. 9:00 – 12:00: GAVL U
5. 12:00 – 12:30: LightGBM

Notable quotes from "Kaggle Solution Walkthroughs: UM - Game-Playing Strength of MCTS Variants with Team James Day"

• So first of all I take the Ludy game rules the essentially the game code and I feed them to a mon car research which does a brief search on the starting position in order to…

  about Tre Search · at 1:01

• Competition organizers this just shows the capus models but I'm pretty sure that the light GBM and tabm models assess the importance similarly so this is a slightly deeper dive into.

  about LightGBM · at 3:36

• Close to 500 extra unique rule sets the extra rule sets were generated via two different means one was I implemented my own version of gavl U which was not open source at the start of this competition and then the other was ordinary.

  about GAVL U · at 9:15

Who and what is mentioned in this episode?

Companies & orgs

• Kaggle — Company · Business focus · 1 mention · first at 0:49
  Platform where the speaker participates in competitions and achieved the Competitions Master title.
  Intermittently for the past few years and I've done well in enough competitions that kagle gave me the competitions Master title quite a while before the start of this competition solution summary so first of all I take the Ludy game.
• University of Maryland — Company · Affiliation · 1 mention · first at 0:39
  Speaker's alma mater where they double majored in computer science and math.
  Machine learning but there's a lot more data wrangling and inference than training academic background I double majored in computer science and math at the University of Maryland and as a hobby I've been working on kagle competitions.

Products & tech

• CatBoost — Product · Case study · 3 mentions · first at 1:30
  Primary machine learning model used in the ensemble for the competition solution.
  Numerical and categorical game features and the CATE Oracle agent features and I feed them to a stacked Ensemble of cat boost light GBM tabm and isotonic aggression models the…
• LightGBM — Product · Case study · 2 mentions · first at 1:30
  Another machine learning model used in the ensemble for the competition solution.
• Isotonic Regression — Product · Case study · 2 mentions · first at 1:36
  Used to rescale raw predictions from initial models to a standardized range.
  Features and I feed them to a stacked Ensemble of cat boost light GBM tabm and isotonic aggression models the isotonic regression models essentially rescale the raw predictions…
• GAVL U — Product · Case study · 1 mention · first at 9:21
  Implemented by the speaker to generate supplemental data for the competition.

Concepts & topics

• Tre Search — Concept · Strategy · 3 mentions · first at 1:18
  Used to compute supplemental game balance features and assess game positions efficiently.

Full transcript with timestamps

0:00 cool um so I'll be presenting my first place solution to the um multicar research strength prediction competition uh the agenda be giving a little background about myself then I'll be giving a summary of my solution then I'll be talking about feature selection and feature engineering then I'll be talking about training methods important findings and a simplified solution so background about myself I'm a software engineer mostly do backend work which involves a small amount of machine learning but there's a lot more data wrangling and inference than training academic background I double majored in computer science and math at the University of Maryland and as a hobby I've been working on kagle competitions intermittently for the past few years and I've done well in enough competitions uh that kagle gave me the competitions Master title uh quite a while before the start of this competition solution summary

1:01 so first of all I take the Ludy game rules the essentially the game code and I feed them to uh a mon car research which does a brief search on the starting position in order to compute a supplemental game balance feature and additional features describing how quickly the Tre search can be run on that particular game then I take those supplemental features and all the ones that most other competitors were using namely the uh numerical and categorical game features and the CATE Oracle agent features and I feed them to a stacked Ensemble of cat boost light GBM tabm and isotonic aggression models the isotonic regression models essentially rescale the raw predictions from the uh from the initial models and um produce basically rescale them to a s range and then both the raw predictions and the rescale predictions are fed to weighted averaging which uh produces the final predictions feature selection and feature

2:02 engineering so this plot here shows the feature importances of the top dozen or so features as assessed by my cap boost models um once one detail that's not too visible in the diagram earlier is that each of those model blocks is really 20 models uh so there's some error bars here where from the models not all assessing the features is equally important um so most important feature all the models agreed about this was the supplemental balance uh metric which is computed by the tree searches and then the next half dozen most important features are essentially the descriptions of the two agents that were playing the games the next most important feature after that was the uh player one advantage metric that was provided by the competition organizers which is computed by uh essentially playing games between random agents that just make random moves that's not quite as valuable of a of a balance metric is the

3:02 one computed by the Tre searches after that it's a couple supplemental features describing how fast the searches can be run those are semi redundant with the features provided by the competition organizer describing how quickly random playouts can be executed after that it's uh a few metrics uh describing the games like the game tree complexity um and there's a bunch more features like that which didn't make it into this plot of the top features and then after that it's the U playouts per second which is provided by the competition organizers um this just shows the capus models but I'm pretty sure that the light GBM and tabm models um assess the importance similarly so this is a slightly deeper dive into the most important features so essentially the balance feature that was computed by the tree searches uh was more strongly correlated with the actual mean agent one utility across all agent

4:05 pairs um for each rle set than uh the advantage P1 that was provided by the competition organizer and that's Illustrated in these Scatter Plots to the right um essentially Advantage P1 versus the mean agent one utility is a very noisy correlation whereas the Tre search based metric it's a less noisy correlation and as a result the models found that the tree search based uh feature was more useful and their uh predictions these partial dependence plots here they show um how the predictions are correlated with the uh treesearch based balance metric and um the models very consistently showed a a linear correlation or a positive correlation but not linear um and they interpolated slightly differently for the tree searches or for the uh decision tree based models versus the interal Network tabm um as for how I selected well oops sorry so there are multiple different Tre search variants that could have been used to compute

5:07 those supplemental features uh the way I decided the configuration to use um kind of went in multiple stages so first I computed those supplemental features with a bunch of different variants and plotted how um their predictions are correlated with um the actual mean agent one you ility across all agent pairs for each rule set and I checked how that scales for different runtimes with different configurations and broadly speaking um the configurations were clustered into good configurations and bad configurations um with the good configurations towards the top of the plot and something that all the good configurations had in common is that they all use some sort of upper confidence bound based selection policy and either random or NST uh play outs um the other settings weren't as important um and then after picking a a couple relatively good configs I then um essentially started feeding features

6:13 from that were computed uh using those configurations to my models and checking how their cross validation scores um basically I checked how well the models scored with features from each different config um something else that's sort of important to highlight here is is that um there's a pretty good correlation even if you only do a search for like 15 or 16 seconds uh more specifically these 16 second searches are roughly as good as one hour of actually simulating matches between uh non-random agents and uh that's on the order of like 200 times uh more efficient um and this is what allowed me to get away with running the tree searches in the submission notebook if if it was uh taking hours per rule set that would not be viable printing methods so I did tenfold group tfold Shuffle cross validation repeated for three seeds typically um I was grouping on the rule set names so no leakage from training

7:13 versus cross validation and the randomization was essentially um randomizing how the rule sets are grouped into folds that's what the random seeds were for um that's something that was not supported by pyit learn prior to this competition although I see that they recently added support for it um and in the submission notebook I used the models from uh two seeds so two times 10 models um the isotonic regression models were trained using out of fold predictions as features um I tuned Ensemble weights uh to in order to um basically minimize root me squared error on my cross validation uh using another me data augmentation so all of the uh tree search based features uh were computed 10 times um and I would intentionally expose my models to a lot of the random variation in those features uh so I train them on uh randomly selected uh feature values from nine runs and hold one run out for cross validation um

8:25 sometimes the machines that were Computing these metrics would run faster or slower and some runs so I found that um it sort of important to hold out one run because none of the training runs would be quite exactly that speed um the random play based features were more computationally demanding to compute so I only recomputed the ones that were provided by the competition organizer once and then I randomly interpolated between the values from the competition organizer versus um the values I computed in order to sort of expose the model the random noise there um then additionally I trained I generated and trained my models on some supplemental data uh more specifically I generated an extra 14,000 rows for uh close to 500 extra unique rule sets the extra rule sets were generated uh via two different means one was I implemented my own version of gavl U which was not open source at the start

9:28 of this competition um and then the other was ordinary off-the-shelf instruction tuned LMS with fot prompting and I also dropped 43 features uh which seemed impacted by Ludy Player version discrepancies uh so to elaborate on that a bit more I observed that when recomputing the features provided by the competition organizer on my machines I got different results consistently for some of the features and I assume that's probably because it's some sort of Ludy player verion discrepancy and I found it beneficial to drop those features so important and interesting findings I think the most important thing to highlight is that the search based game balance feature was extremely useful that improved my scores on the leaderboard by close to 0.01 or a bit over 0.01 and uh that's what initially got me to first place on the public leader board and um is what probably what allowed me to hold on to the first

10:29 place spot I think that some of the other competitors had better data augmentation strategies than me um but this supplemental theature was beneficial enough to outweigh that additionally those supplemental features are pretty computationally efficient to compute um which is what allows me to get away with running with generating them um another thing I'd like to highlight is that the supplemental training data I generated was only marginally useful um there was a slight Improvement in the third decimal point on the leader board both cross validation and leader board um and I suspect the reason this wasn't very useful was that it consumed an egregious amount of CPU time in order to annotate these supplemental grows in order to save CPU time I tried reducing the number of matches played per agent pair which caused the labels to be rather dirty um with the benefit of hindsight I wish I had added half as many rows with

11:26 double the compute per row um just to make the labels less noisy simplified model so with only my cat boost models and the isotonic postprocessing I can score0 421 on the private leaderboard and that's good enough for fourth place and additionally roughly half of the 600 or so features are not very important and can be dropped with only a very slight score degradation um of 0.001 and this plot here shows those is it's from a fairly early experiment with light GBM where I was checking how the cross validation error scales with the number of drop features and I found that dropping the unimportant features uh was not beneficial um but it didn't really cause a huge amount of harm either I can drop half of the feature with only a very slight degradation okay uh Q&A do you guys have any questions for me

Questions about this episode

Who speaks in "Kaggle Solution Walkthroughs: UM - Game-Playing Strength of MCTS Variants with Team James Day"?

James Day (Host).

What is "Kaggle Solution Walkthroughs: UM - Game-Playing Strength of MCTS Variants with Team James Day" about?

The content presents a first-place solution to a machine learning competition focused on predicting the game-playing strength of Monte Carlo Tree Search (MCTS) variants. The discussion covers the author's background, methodology for feature selection and engineering, as well as the training techniques and key findings behind their successful predictive model, which combines various machine…

What is mentioned in this Kaggle episode?

7 people, works and ideas are mentioned, including Tre Search, CatBoost, LightGBM, Kaggle, Isotonic Regression, University of Maryland. Each one on this page links to its own page with every episode that mentions it.

How long is "Kaggle Solution Walkthroughs: UM - Game-Playing Strength of MCTS Variants with Team James Day"?

The episode runs 12 min; the transcript is about 1,892 words in 12 timestamped sections.

More from Kaggle

• All analyzed episodes
• People mentioned on Kaggle
• Topics mentioned on Kaggle