Kaggle Solution Walkthroughs: Eedi - Mining Misconceptions in Mathematics with Team CQYR

What this episode is about

The topic focuses on the solution developed by Team CQYR to address misconceptions in mathematics using machine learning techniques as part of a Kaggle competition. Key points include the team's introduction, their background in HR tech and machine learning, an overview of their five-phase approach to data generation and training, and their strategy for leveraging existing educational datasets to enhance the model's contextual understanding.

Key takeaways

• Mistakes are teachers; leveraging Chain of Thought transforms errors into learning opportunities for models.
• Larger models may be cliché, but they really do pack more punch—size matters in machine learning!
• Synthetic data: the unsung hero! It boosted our scores by 7K points—talk about artificial intelligence!
• Misconceptions are fertile ground for innovation; enriching them can enhance understanding and model performance.
• Incorporating contextual parent subjects? It's like giving your models a family tree—much richer insights emerge!

Who speaks in this episode?

• Lehan (Host) · 3 min speaking
• Ywa (Guest) · 1 min speaking
• Jan (Guest) · 1 min speaking
• Okasa (Guest)
• Okama (Guest)

What is discussed, and when

1. 3:00 – 6:00: Quin
2. 6:00 – 9:00: Quin
3. 9:00 – 12:00: Quin
4. 12:00 – 13:43: Auto R, Auto AWQ, Auto GPTQ

Notable quotes from "Kaggle Solution Walkthroughs: Eedi - Mining Misconceptions in Mathematics with Team CQYR"

• Here I'm sorry only leaderboard score here and so basically you can see the larger the model size the better the score here and the inference time also increase and as for the training of the rank we use the QA meths and the base.

  about Quin · at 11:47

• Is the it is easy to use and we think it is the most user friendly one and also it can maintain the superior accuracy after the quantization here is the appliation study and so I will sorry I will skip.

  about Auto R · at 13:10

Who and what is mentioned in this episode?

Products & tech

• Quin — Product · Backbone model · 3 mentions · first at 3:02
  Used as the primary model for generating explanations and handling chain of thought in the solution
• Auto R — Product · Quantization tool · 1 mention · first at 13:06
  Selected as the preferred tool for post-training quantization due to its user-friendliness and accuracy
• Auto AWQ — Product · Quantization tool · 1 mention · first at 13:01
  One of the tools considered for post-training quantization to optimize model inference
  Training data as a calibration data set and we have for the quantization to tools we have tried Auto gptq auto awq and auto R and finally we choose auto run because it is the…
• Auto GPTQ — Product · Quantization tool · 1 mention · first at 13:01
  One of the tools considered for post-training quantization to optimize model inference
  Training data as a calibration data set and we have for the quantization to tools we have tried Auto gptq auto awq and auto R and finally we choose auto run because it is the…

Full transcript with timestamps

0:00 everyone my name is lehan I and I will present our solution today and sorry and uh first the agenda let us first introduce our background and then we go to the summary of our solution and after the summary we go a little bit deeper into the detail of the future selections and the training part of our Solutions so first part the background uh yeah of our members live and work in Japan so let's first start from ywa y could you please introduce yourself a little bit hi my name is name is I'm working in HRT company as a machine engineer okay okay okay thanks and uh the second one song can you please okay everyone uh my name is Jan I also work at the HR tech company I'm a machine engineer almost uh focus on recommendation system that's all thank you okay thanks and okasa has some urgent issue and he is not here today and he is my ex colleague and he works as a machine learning engineer at IT consulting firm

1:31 and the me my name is lean I'm also working as a machine learning engineer at a tech company and this is my company in the background name and uh yeah actually it it is the second NLP competition for me and okama we both F uh majorly focus on computer vision and so let's start our solution and first summary and our solution basically composed of five phases three for data generation and the two for training and the first one is just a simple preprocessing uh we in the early stage we found some useful information from the past Ed computation there is a subject matter CSV and with subject ID and subject name which is identical to the one in this competition so I believe they come from the same source and the subject metadata contain some parents subject like uh number for example number geometry and measure statistics so I just add this parent subject to the training and the tested data to give some more contextual

2:50 information and this is the first phase and the second one is Chain of Thought pretty much like the first place but we did not perform extra fine tuning just use the pre-trained Quin model and here is the prompt we used and this is the key part of the prompt we make the model we fed the model the WR answer and the question text to make it gener to make it explain the mistakes step by step and then described the misunderstanding behind the wrong answer to generate some information here and the SEC the third one is also pretty much common that is ined data I believe most of the teams use this technique and we uh we also generated many synthetic questions we generated three rounds of synthetic questions the process is pretty much the same we first fed the model the misconception name and some few short examples to make it generate a question text and after the question text is erated we fed that question text

4:02 to the mass models to let the mass models solve the question and give the wrong answer and the correct correct answer and after this is done we fed all of this to the chat GPT to let it check the quality of the question text whether it is a good Destructor or Not by soring the uh the question and besides the synthetic questions we also made another ex generation I call it misconception augmentation which is to enrich the misconception informations you know that uh the misconception basically contains only one short sentences and let let we just think that if we can generate some more text about to describe the misconceptions that will boost the score in the retrieve and uh yeah here is an example we let the model to explain the misconception shortly and provide some short cases here like this and after this three phases all of the data are done and we go into the training part and the first training

5:18 part is the is the retrieve training um the key factors is as listed here well as for the model basically the larger the better and for models like qu they are not um retre models but we used to V library to enable the bidirectional attention instead of the default casual attention to make it fit for the retrieve task and uh for the retrieve I think the most important features is the misconception augmentation which boost the score for two about two 2 point to Four Points here and also synthetic data play a very important role here adding about 7K sorry 7K high quality syet data boost score lot and besides Chain of Thought and also the Ping selection here we found that the last token pudding performs better than the mean puding and that's for the retrieve and for the rerank the key factors is pretty much the same basically the the larger model performs better and uh the high syn the high quality syn dat also works

6:44 and the most important feature here is a chain of s and and we also randomly changeed the negative numbers when during the training to make the models more robust and for the inference uh we used the listwise with sliding window methods and TTA and also performed the post training quantization to make it work in the carago inference inference in invironment and here is an image of our ranking process we first rank the E to 17 candidates of the the candidates provided by the retrieve and after this rerank is done we rerank the top 10 here to enable that the correct answer in a lower rank will be pushed up to the higher rank and this is the summary and uh let's go a little bit deeper into the feature selection and Engineering so the first one is the chain of s and the yeah we use Quin as as as a backbone for the chain of s and let the model explain the mistake and the then describe the misunderstanding and actually we have an

8:16 experimental V2 experiment but we just didn't use this because of the time constraint the main issue is to deal with the questionss the name like Tom because we find that when the question text contains names like Tom or KY the M uh the model will Mis mistakenly try to explain why Tom or KT is wrong or right that's not the purpose so we add some trick to deal with that and also add a self reflection base to make a more clever Chain of Thought here I have attached it in the appendix and so if you are interested you can take a look at that and the next one is the synthetic data so here's the prompt of how we generated the miscon the explanation for each misconception here so this is a key part of the prompt and please explain the misconception in detail and provide some short cases when the misc conception will occur which leads to the pro which leads to the example I showed above and uh and

9:39 for the synthetic questions we have generated synthetic questions for three rounds the first two rounds is quite similar as the process I showed up before but the third generation is a little bit tricky um actually we Pro we randomly sampled two questions from the training data and the synthetic data generated in generation one and generation two and in in generation three we let the model fuse two questions with same misconception to generate a more difficult difficult questions and here is the example of the prompt and the key instruct is here we let the model add some steps to make the question more difficult and also instructed to change the context of the question and also add a self reflection phase to ensure the quality of the question and here are some do and don't here and as for the Training Method the first part is the retrieve training and we have two pipelines here the first one is developed by me and the

11:02 second one is mainly developed by ywa and uh the f I think we basically follow the constructive learning and uh the difference is that in pipeline one we used arace loss and the second one we used the multi negative ranking loss here and as for the big models we didn't use chain ofs here because if we add the chain of Sting to The Prompt the Imp time will increase significantly which will cost time out in the cargo environment and uh here is the cross validation and the leaderboard score here I'm sorry only only leaderboard score here and uh so basically you can see the larger the model size the better the score here and the inference time also increase and as for the training of the rank we use the QA meths and the base models are quen and lmer and uh so basically the larger the model size the better in Cross validation score and the private leaderboard but one thing which we felt a littleit bit

12:28 conf confused is that the large model like 72 billion does not show the Improvement in the public leader board and for and for the rankers we performed the post training quantization to make it faster in the inference and we first merged the ca into the base model and performed quanti using the training data as a calibration data set and we have for the quantization to tools we have tried Auto gptq auto awq and auto R and finally we choose auto run because it is the it is easy to use and we think it is the most user friendly one and also it can maintain the superior accuracy after the quantization uh here is the appliation study and uh so I will sorry I will skip this slide here and thank you very much and any questions

Questions about this episode

Who speaks in "Kaggle Solution Walkthroughs: Eedi - Mining Misconceptions in Mathematics with Team CQYR"?

Lehan (Host), Ywa (Guest), Jan (Guest), Okasa (Guest), Okama (Guest).

What is "Kaggle Solution Walkthroughs: Eedi - Mining Misconceptions in Mathematics with Team CQYR" about?

The topic focuses on the solution developed by Team CQYR to address misconceptions in mathematics using machine learning techniques as part of a Kaggle competition. Key points include the team's introduction, their background in HR tech and machine learning, an overview of their five-phase approach to data generation and training, and their strategy for leveraging existing educational datasets…

What is mentioned in this Kaggle episode?

4 people, works and ideas are mentioned, including Quin, Auto R, Auto AWQ, Auto GPTQ. Each one on this page links to its own page with every episode that mentions it.

How long is "Kaggle Solution Walkthroughs: Eedi - Mining Misconceptions in Mathematics with Team CQYR"?

The episode runs 14 min; the transcript is about 1,700 words in 10 timestamped sections.

More from Kaggle

• All analyzed episodes
• People mentioned on Kaggle
• Topics mentioned on Kaggle