Kaggle Solution Walkthroughs: UM - Game-Playing Strength of MCTS Variants with Team Richard_U

What this episode is about

The main topic is a detailed walkthrough of the methods and processes employed by Team Richard_U in a Kaggle competition focused on evaluating game-playing strengths of various Monte Carlo Tree Search (MCTS) variants. Key points include background in statistics, a straightforward model using a weighted ensemble of boosting models and neural networks, effective data augmentation techniques, and strategies for cross-validation and leaderboard optimization.

Key takeaways

• Data augmentation: not about perfection, but about strategic imperfection that boosts model performance.
• Flipping agents can flip outcomes—an elegant trick that uncovered hidden patterns and improved scores.
• Mastering the mundane: meticulous data prep and clever grouping led to surprisingly better cross-validation results.
• In ensemble learning, sometimes six heads are better than one—variety in models truly enhances predictions.
• With a mere three days of training, the right mix of data and models can lead to competitive breakthroughs.

Who speaks in this episode?

• Richard (Host) · 14 min speaking

What is discussed, and when

1. 0:00 – 3:00: Cross Validation, Feature Engineering, Public Leaderboard, Kaggle
2. 3:00 – 6:00: Cross Validation, Feature Engineering
3. 6:00 – 9:00: CatBoost, Cross Validation
4. 9:00 – 12:00: Cross Validation, CatBoost, LightGBM, Public Leaderboard
5. 12:00 – 15:00: Public Leaderboard, Cross Validation

Notable quotes from "Kaggle Solution Walkthroughs: UM - Game-Playing Strength of MCTS Variants with Team Richard_U"

• Income comes from Financial assets trading with my own private capital and I have participated in seven kaggle competitions before this one with the incentive to develop my…

  about Kaggle · at 0:40

• Wading based F50 directly from the public leaderboard I get back to that as well okay data preparation and feature engineering I start with the data augmentation which I think was the thing oh single most.

  about Feature Engineering · at 2:10

• Like could be expected almost strictly increasing framing methods the overall submission was a an assemble of seven models four boosting models and three Neal Network models the boosting models train with and without the d r set.

  about Cross Validation · at 8:13

• Train with the, 1400 is Dam Ru set extra dam is and without and then I blended the cat boost and the light key bam models with an 8020 blend I probably could have gone with…

  about CatBoost · at 9:01

• That was what I just described that was my cross validation approach and then I also did a public leaderboard fitting approach which was identical but with two differences in the public leaderboard fitted approach I also included.

  about Public Leaderboard · at 12:00

Who and what is mentioned in this episode?

Companies & orgs

• Kaggle — Company · Business focus · 1 mention · first at 0:47
  Platform where the competition took place, central to the episode's narrative.

Products & tech

• CatBoost — Product · Case study · 2 mentions · first at 7:14
  Used as part of an ensemble model in a Kaggle competition, contributing to the winning submission.
• LightGBM — Product · Case study · 1 mention · first at 9:06
  Used alongside CatBoost in the ensemble model for the Kaggle competition.
  Without and then I blended the cat boost and the light key bam models with an 8020 blend I probably could have gone with just cat boost and skip the light key bams but because they didn't seem to add much not in the not in Cross validation.

Concepts & topics

• Cross Validation — Concept · Strategy · 9 mentions · first at 1:14
  Key technique used to validate model performance in the competition.
• Feature Engineering — Concept · Core topic · 3 mentions · first at 0:06
  Central to the discussion of data preparation and model performance improvement.
• Public Leaderboard — Concept · Key argument · 6 mentions · first at 0:11
  Discussed as a metric for model performance and potential overfitting in the competition.
• Monte Carlo — Concept · Framework · 1 mention · first at 2:54
  Mentioned in the context of data augmentation and model training strategies.
  I looked at the data and I saw that for the Monte Carlo research algorithm settings when agent one had certain specifications for a certain game and agent two had others in the…

Full transcript with timestamps

0:00 of the slides quite quick uh almost the the same agenda background summary data preparation and feature engineering training methods public leaderboard fitting and finally simple model background I have studied quite a lot and I have several degrees but of relevance to this competition is a master in statistics I have a varied professional background uh ran my own company for many years uh and they have done a lot of other things but for the last five years my income comes from Financial assets trading with my own private capital and I have participated in seven kaggle competitions before this one with the incentive to develop my data science and statistical skills and I have learned a lot from all of them and it was was a lot of fun OKAY model summary I would overall I would describe it as a straightforward approach uh five split cross validation data augmentation I will get back to the data augmentation

1:18 in a separate slide I use no additional offline generated training data and uh no feature selection at least no feature elction to speak of and uh the overall model consisted of an weighted Ensemble of seven models where of four boosting and three Neal networks and the final submission used an equally weighted mix of six full sets of models where the used seed for training was the only difference and to Total training time was less than three days on one machine and I actually had two final submission one cross validation based approach and another one almost exactly the same using the the same trained models but with some additional wading based F50 directly from the public leaderboard I get back to that as well okay data preparation and feature engineering I start with the data augmentation which I which I think was uh the thing oh single most important uh data preparation and feature engineering I did and after the

2:37 competition I saw that several other teams used exactly the same augmentation which did not surprise me because I think it was given the data quite natural approach what I did was that uh I looked at the data and I saw that for uh the Monte Carlo research algorithm settings uh when agent one had certain specifications for a certain game and agent two had others in the data the uh inverse was not present with flipped agents so I thought what if we just flip the agents well if we flip the agents then probably we should also Flip Flip the Target because if agent one performed well then yes so so sign flipping of the target seemed reasonable and if we flip the agent setting and we flip the target then probably we should flip this important variable Advantage play one which uh doesn't seem to perfectly capture the play imbalance in the game but it it it gives still gives quite a lot of information and flipping this

3:49 variable would be taking one minus the value because the range of Advantage player one was between zero and one and then we come up with some kind of up side down version of the game or mirror version or something which is uh this augmentation is by no means perfect I would say but on the other hand the idea of augmentation is not to make an absolutely perfect representation because then we don't add anything or at least don't add much but it should be it the augmentation should add something which hopefully improves the model performance and uh this significantly improved both cross validation and public leader B score and I also added an augmented data D feature to let the models know which rows of data that originated from augmentation and I finally I also used augmentation in inference so all the models were trained on both the original data set and the augmented set and uh at inference I made

4:53 two predictions one from the original test data and one from uh the augment and mod modified test data other data preparation and feature engineering I will go through this quick because I don't think it's that interesting uh dropped duplicate and non information features calculated text scores features from the lad rules column and I want to thank kagle profile Yun for this idea I didn't come up with this myself i s data in groups created from the L RS column by tfidf vectorization and K clustering thus reducing the the number of rule sets and the reasoning behind this was to uh make cross validation estimates that hopefully generalized better outside the training data if similar games were grouped together in in the same fold and I I I think this I don't have any numbers but I'm pretty sure this improved the performance a little bit uh I also D encoded unique Ru sets uh from the L rules column these were

6:08 not used in the ne Neal Network models because it didn't seem to improve performance but it seemed to uh slightly make the the boosting models generalize better so I trained half of the boosting models with this approximately 1,400 extra Dumis then I dropped the ruls columns and ID use data augmentation as discussed in previous slide I one encoded the uh agent specification category of features with the most common value used as Baseline so in in in the training data there were only dummy features and continuous variables and the continuous variables were mapped to standard normal distributions before training of and interference from the network Neal Network models here's a feature importantance plot from a catus model trained on all the data I'm not going to comment this much uh except that uh there's there's a lot of these U agent settings uh dumy features present in the top 20 features which is

7:25 nice to see because sometimes when I was doing this competition I was think oh are we mainly just predicting the game characteristics characteristics and not so much uh the the performance of the the diff different algorithm settings but as you can see it seems that they play quite a big role actually which is nice to see I think and this is extended fli feature here that's my augmented data dumy and I think this indicates the non perfectness if you will the data orentation that is quite important in the model here's a partial dependence plot of the advantage play one also from an a cat boo model train with all of the data it looks like could be expected almost strictly increasing framing methods uh the overall submission was a an assemble of seven models four boosting models and three Neal Network models um the boosting models train with and without the d r set identifiers and I uh then weighted these

8:40 seven models using out of fold predictions and the an ordinary least squares regression and to not get too many parameters to fit and in the regression I first Blended the the boosting model predictions with 50/50 Blends uh of uh the dam with a model train with the, 1400 is Dam Ru set extra dam is and without and then I blended the cat boost and the light key bam models with an 8020 blend I probably could have gone with just cat boost and skip the light key bams but because they didn't seem to add much not in the not in Cross validation scores and not in the public leader board but I used to include them with 20% hopefully we' hope that they would generalize better and this Blended boosted prediction predictions came in in U in in in two versions one using the original test data and one using the augmented modified test data and then three neuronetwork models the standard multi-layer percep the second just

9:57 bigger multi-layer percep and finally an outo encoder Follow by mulay perception and the outputs from the Nal networks were used directly in the the squares waiting and here are just uh the waiting estimates uh you can ignore the P values and confidence intervals uh the uh aside from selection bias Etc the the the data is of course also heavily clustered in the different games but it it shows that uh the boosting models of uh big importance the augmented predictions on the boosting models big importance the negative sign here is is just because uh negative sign were expected for the augmented predictions and the the first n model important and unexpected for me at least was that the bigger multilayer perception and the out encoder gave negative Ensemble estimates uh generally I'm I'm I I'm quite skeptical to to negative waiting but it it improved both the cross validation and public leaderboard

11:24 scores I decided to keep them uh and probably it's just because they're overfit but I think with more time I probably could just adjusted the settings and got positive estimates and maybe a better model and I also included I want topl one with a minor waiting I don't think it it did much but maybe it reduced overfitting a little bit or something and then I kept the alound prediction to the target range and scaled them again as slightly linear upscaling and then finally kept the final predictions that was uh what I just described that was my uh cross validation approach and then I also did a public leaderboard fitting approach which was identical but with two differences in the public leaderboard fitted approach I also included augmented data predictions from the Nal Network models and uh they got negative weights which I fitted directly from the public leaderboard and secondly I also include a prediction

12:33 set with equal weights to all model predictions but sign flipped we expected that is the augmented data predictions and these total these total set predictions were then added with again negative weights fited directly from the public leaderboard and I I interpreted this when I did it towards the end of the competition I interpreted this as as a almost desperate obvious public leaderboard overfitting approach but uh of course it improved the public leader board score since uh the the way thing was fitted from the public leader board but it also improved the private score and after the end of the competition there were some complaints uh regarding the private uh test set and and I think that's to be expected in almost every competition that there would be complaints about the constitution of the priate test Set uh and I I do think that my cross validation based submission would get a fairly decent ranking among other cross

13:35 validation based submissions but still I do think that I also want to say that I absolutely understand the hosts argument why they choose to constitute the private test as they did but I still think there more generally speaking I think there there is some credit to to uh the argument that ideally uh the should not be a much stronger alignment between the private test set and the public test set than between say the training data and the public test set or for that matter the training data and the uh the public test set yeah I just want to mention that and finally a simple model I go through this very quick cuz I I have this got feeling that I'm over my time uh I first started a simple model with the top top 10 features it performed really badly so I extended to the top 20 including the most uh important agent settings uh and a single catus model trained on all the training data with top 20 features gave a private score of

14:53 0.48 compared to around 0.42 for my second place uh winning model uh and given that this single model only used 20 features and one model compared to this kind of monstrous total model numbers for my winning submission I think it's a decent score uh and for comparison I also added an ultra simple just linear fit using the advantage play1 uh column alone and that would give a private score of 0.55 okay thank you

Questions about this episode

Who speaks in "Kaggle Solution Walkthroughs: UM - Game-Playing Strength of MCTS Variants with Team Richard_U"?

Richard (Host).

What is "Kaggle Solution Walkthroughs: UM - Game-Playing Strength of MCTS Variants with Team Richard_U" about?

The main topic is a detailed walkthrough of the methods and processes employed by Team Richard_U in a Kaggle competition focused on evaluating game-playing strengths of various Monte Carlo Tree Search (MCTS) variants. Key points include background in statistics, a straightforward model using a weighted ensemble of boosting models and neural networks, effective data augmentation techniques, and…

What is mentioned in this Kaggle episode?

7 people, works and ideas are mentioned, including Kaggle, Cross Validation, Feature Engineering, CatBoost, Public Leaderboard, LightGBM. Each one on this page links to its own page with every episode that mentions it.

How long is "Kaggle Solution Walkthroughs: UM - Game-Playing Strength of MCTS Variants with Team Richard_U"?

The episode runs 16 min; the transcript is about 2,057 words in 13 timestamped sections.

More from Kaggle

• All analyzed episodes
• People mentioned on Kaggle
• Topics mentioned on Kaggle