Kaggle Solution Walkthroughs: Eedi - Mining Misconceptions in Mathematics with Team Waseda Pochi

What this episode is about

The presentation by Team Waseda Pochi details their innovative two-stage retrieval architecture designed to address misconceptions in mathematics. Key components include an embedding model for efficient retrieval and a ranking model for improved performance, enhanced by a novel post-processing method that specifically addresses unseen misconceptions, leveraging advanced training techniques and ensemble strategies.

Key takeaways

• A rabbit-inspired team from Taiwan and Japan proves that even the cutest mascots can deliver data-driven solutions.
• Two-stage retrievals aren't just trendy—they're crucial when 75% of unseen data could tank your predictions.
• Training a ranking model with misconceptions first? That's a genius misstep, costing you precious performance points!
• Ensemble models: because why settle for one good output when you can juggle several mediocre ones?
• Post-processing isn't just a phase; it's the secret sauce that can spice up your competition score by 0.03.

What is discussed, and when

1. 0:00 – 3:00: Q 2.5 LM, AI Square lab, Kahala lab, National Taiwan University
2. 6:00 – 9:00: cargo

Notable quotes from "Kaggle Solution Walkthroughs: Eedi - Mining Misconceptions in Mathematics with Team Waseda Pochi"

• Findings and first is on the background of our te oh I am e and I'm a master student in computer science and I'm from Taiwan and now study in National Taiwan University and I'm…

  about National Taiwan University · at 0:21

• Solution our solution is a two stage retrieval architecture just like most of the other team and the stage one is it's a embedded model based on the Q and 2.5 LM and the stage two is.

  about Q 2.5 LM · at 1:14

• Then it's our post processing method and we and by looking the cargo discussions discussion we discover that there are some unseen subject in the testing data and so we do some lead prob and we find that when.

  about cargo · at 7:28

Who and what is mentioned in this episode?

Companies & orgs

• cargo — Company · Platform · 1 mention · first at 7:33
  Platform used for discussions and competition data
• National Taiwan University — Company · Affiliation · 1 mention · first at 0:34
  University where the speaker is currently studying
• Wasa University — Company · Affiliation · 1 mention · first at 0:43
  University where the speaker is pursuing a double degree
  Student in computer science and I'm from Taiwan and now study in National Taiwan University and I'm a double degrees in a double degrees program with Japan Wasa University and…
• AI Square lab — Company · Affiliation · 1 mention · first at 0:47
  Lab where the speaker is a member in Taiwan
  University and I'm a double degrees in a double degrees program with Japan Wasa University and I'm the member of AI Square lab in Taiwan and Kahala lab in Japan and the name of…
• Kahala lab — Company · Affiliation · 1 mention · first at 0:50
  Lab where the speaker is a member in Japan
  Double degrees program with Japan Wasa University and I'm the member of AI Square lab in Taiwan and Kahala lab in Japan and the name of my team is after my killed rabbit the rabit in this photo and the rabit in brown and.

Concepts & topics

• Q 2.5 LM — Concept · Core technology · 1 mention · first at 1:36
  Central to the solution architecture, used in both stages of the retrieval model

Full transcript with timestamps

0:00 oh hello everyone we are the third place team Wasa po first this is the agenda for our presentation there are four parts from background solution overviewed and Training Method and some important findings and first is on the background of our te oh I am e and I'm a master student in computer science and I'm from Taiwan and now study in National Taiwan University and I'm a double degrees in a double degrees program with Japan Wasa University and I'm the member of AI Square lab in Taiwan and Kahala lab in Japan and the name of my team is after my killed rabbit the the rabit in this photo and the rabit in brown and white and then is the solution the overview of our solution our solution is a two stage retrieval architecture just like most of the other team and the stage one is uh it's a embedded model based on the Q and 2.5 LM and the stage two is uh ranking model based on also based on Q 2.5 LM but it's a bigger

1:44 version and our training is mostly most most mostly based on K of if for efficient training and inference and we use some model emble strategy to improve our performance and and the most special point of our solution is the noal postprocessing method for for unseen with conception handling and then let's go into the training methods part and first first the stage one is the retrieval part our model is uh Q 2.54 billion instruction based on this LM and it's and it's quized into 4bit with 8 up q and we finding finding this model as to EMB better with the famous in Bedding Library flag in that flag in that bedding and this is the Ur URL of the library and on the left on the right hand side this this is our model architecture we inut we inut misconceptions and question into the Eder and get the embedding of them and and calculate the cine similarity and in stage one we we retrieve the top 35

3:32 mtion and put it and into input it into the stage two and we do some Ensemble on this model we train four different four different inor um they are three TR with competition data only but with different random seed so they will generate slide um similar but different result and one is TR with some gbd for mini synthetic data but with the synthetic data the model didn't show showing performance Improvement but we also include it into the on mod and the Standalone performance of our St stage one model is 0.4 M4 in per lebard and in in the final vers final our best version we also assemble the best public Invader best public Invader model on cargo and then it's our stage stage two model St ring model and of St model is also based on Qin but it's Qin 2.5 32 billion instruct instruct model and it's also quized in into four bits with a up Cube and we and we only find and for the stage two data we only use

5:20 the competition data and without s any other synthetic or external interal data set and and in the left hand in the right hand side is the our model architecture it's just like other chat chat loging model which inut the promp promp with uh question uh correct answer and wrong answer and also input the possible misconception retrieve in stage one and there are total totally 35 35 misconception and we assign un ID is to each misconception and then ID number is from one to n and A to Z or there are sum to 35 and and then we ask the loging model to Output which one of the best which s of the misconception is the best and we train this model with super advis fight unit with cor Central v l and finally when iner time we will we will we will get us the probability of each each over probability of each ID and use the probability as the score of each misconception and we assemble six model which TR r with different R

7:09 seed and the the performance of our stage two model is 0.57 it's it's uh it's my rank at maybe 10 and it's already in gold medal area and then it's our post processing post processing method and we and by looking the cargo discussions discussion we discover that there are some unseen subject subject in the testing data and so so we do some lead prob and we and we find that when we estimate that they are maybe there are about 75% of the Ving data the the answer of the testing data is um is unse Mis compensation in the in the training data it's mean that we don't have the ex example of this misconception in the competition training data and so our appro is quite simple just just introduce the scoring factor C for the on onse misconception scores and we just multiply the probability the score of the on misconceptions in training training data set and dyam Dally adjust the C until the they are

8:57 about 75% of the top one prediction on were unseen misc compession on the on whole of the testing whole the prediction of testing data set and on we we we assum that this aligned our prediction with the T distribution and this meth give us 0.03 Improvement and finally our score is about 0.6 0.6 and it's worth to mention that we also this this matching approach will also explored by other top team side and then is the the in fighting spot and one of the interesting fighting of of our of our experiment is that when training the ranking model there are big performance Gap when you when the P was question first or misc conversation first first and he said and at the beginning we we saw that for for time for optimize the efficiency see we can we can use put the misconception first in the pump so we can reuse the K cach across different question but however when we train the model with the promp was misc comption

10:47 first first the score is the score disc the score is slower about 0.1 compared with the question first prom uh we think this is a quite interesting um observation and but but we we don't have but we don't have enough time to to analyze it deeper and also there are many methods we tried by that work for expand for the c pomping or or reite the visc conception and thanks to the Ed group and kago for organizing this competion the try also this turn not work it Hur me a lot okay so thank you thank you for listening

Questions about this episode

What is "Kaggle Solution Walkthroughs: Eedi - Mining Misconceptions in Mathematics with Team Waseda Pochi" about?

The presentation by Team Waseda Pochi details their innovative two-stage retrieval architecture designed to address misconceptions in mathematics. Key components include an embedding model for efficient retrieval and a ranking model for improved performance, enhanced by a novel post-processing method that specifically addresses unseen misconceptions, leveraging advanced training techniques and…

What is mentioned in this Kaggle episode?

6 people, works and ideas are mentioned, including Q 2.5 LM, cargo, National Taiwan University, Wasa University, AI Square lab, Kahala lab. Each one on this page links to its own page with every episode that mentions it.

How long is "Kaggle Solution Walkthroughs: Eedi - Mining Misconceptions in Mathematics with Team Waseda Pochi"?

The episode runs 12 min; the transcript is about 1,159 words in 7 timestamped sections.

More from Kaggle

• All analyzed episodes
• People mentioned on Kaggle
• Topics mentioned on Kaggle