Why High Benchmark Scores Don’t Mean Better AI [SPONSORED]

Who speaks in this episode?

• Andrew Gordon (Guest)
• Nora Petrova (Guest)

What is discussed, and when

1. 0:00 – 3:00: Prolific, Andrew Gordon, Nora Petrova, Grock 4
2. 3:00 – 6:00: Chatbot Arena, Anthropic
3. 6:00 – 9:00: Chatbot Arena, Llama 4
4. 9:00 – 12:00: Chatbot Arena
5. 12:00 – 15:00: Prolific, Chatbot Arena

Notable quotes from "Why High Benchmark Scores Don’t Mean Better AI [SPONSORED]"

• My name is Nora Petrova. I'm an AI researcher at Prolific. I'm tackling questions around how do we include humans development of AI models and evaluation of AI models? How do…

  about Prolific · at 0:45

• Values are. What would be a fairer approach to understand the behavior of models and to do evaluation? So this is what Andrew and Nora have been doing at prolific. How do we…

  about Andrew Gordon · at 2:07

• How fast or smart the model is. how safe is it for the people to use? There's been a lot of interesting research coming from anthropic in that direction with regards to safety…

  about Anthropic · at 4:54

• Able to handle novel situations in safe ways. I think the one other thing I'd say is given that chatbot Arena is the number one and frankly pretty much the only human…

  about Chatbot Arena · at 5:43

Who and what is mentioned in this episode?

People

• Andrew Gordon — Person · Main speaker (guest) · 1 mention · first at 0:32
  Introduces himself as a Prolific staff researcher and argues that technical benchmarks miss real-world human user experience; explains Prolific’s human-preference evaluation approach and its methodological choices.
• Nora Petrova — Person · Main speaker (guest) · 1 mention · first at 0:45
  Frames the need to include humans in AI development/evaluation and describes the statistical/methodological backbone (TrueSkill, information gain sampling, demographic slicing) used for Prolific’s comparative model battles.
  My name is Nora Petrova. I'm an AI researcher at Prolific. I'm tackling questions around how do we include humans development of AI models and evaluation of AI models? How do…

Companies & orgs

• Prolific — Company · Business focus · 4 mentions · first at 0:34
  Primary company discussed in the podcast, focusing on human-centered AI evaluation and research.
• Anthropic — Company · Referenced expert · 1 mention · first at 5:02
  Mentioned for their research in AI safety and model alignment.

Products & tech

• Chatbot Arena — Product · Case study · 4 mentions · first at 5:47
  Discussed as a leading human preference leaderboard for LLMs, with critiques on its methodology.
• Grock 4 — Product · Referenced work · 1 mention · first at 2:46
  Mentioned as an example of an AI model with emphasis on technical benchmarks over user experience.
  Some pretty starking examples recently with Gro 3 and Mecca Hetler and it does raise questions about how thin of a veneer is the safety training on top of some of these models.
• Llama 4 — Product · Referenced work · 1 mention · first at 6:17
  Discussed in the context of model evaluation and the integrity of leaderboards.
  More private testing in the background than others. For instance, before Llama 4 launched, we saw Meta released 27 models on the arena. But of course, only one was actually…

Full transcript with timestamps

0:00 Formula 1 cars are the absolute pinnacle of engineering, right? Everything is perfect, huge top speed. If you used it as your daily commuting car, you'd have an absolute nightmare, right? And I think the same can be said for these models, right? A model that is incredibly good on humanity's last exam or MMLU might be absolute nightmare to use day-to-day. Most reporting on benchmarks is done on technical benchmarks these days, right? Um that is where you get the model uh you you give it a set of evaluations uh maybe on one theme or maybe on an exam and then you get a score and humans aren't really involved in that loop. Hey, my name is Andrew Gordon. Uh I'm a staff researcher in behavioral science at Prolific. So I work on the sciences team uh tackling questions related to humans in research specifically online research.

0:45 My name is Nora Petrova. I'm an AI researcher at Prolific. I'm tackling questions around how do we include humans development of AI models and evaluation of AI models? How do we align them to human values? And how do we fully understand what they're capable of?

0:59 How helpful do people find models? What's the communication like? How adaptive do they find it? What do they think of the model's personality? And that when you get ratings on those kind of factors, what you actually get is an actionable set of results that say, okay, your model is struggling with trust or your model is struggling with personality. And we had a first go at this with what's called the prolific user experience leaderboard. Uh that was kind of proof of concept with 500 participants from the US uh a representative set of participants where they evaluated a single model at a time and gave their feedback in a like scale format. You know how helpful did you find the model 1 to 7 and so on. We've actually moved now and taken the learnings from that initial leaderboard and and built it into what we're calling humane which is our main leaderboard. Um and that uses a much a similar approach

1:47 to chat arena in the sense that we have comparative battles between um between models uh which actually allows us to much more clearly differentiate which model is performing better. What if we could have a a fairer approach where we actually diversely sampled and stratified folks based on how old they are and where they live and what their values are. What would be a fairer approach to understand the behavior of models and to do evaluation? So this is what Andrew and Nora have been doing at prolific. How do we know whether these models are actually good for the humans that are using them? How can we make evaluation metrics which are fairer?

2:24 This is Andrew and Nora. But the problem is at the moment I mean the the field of kind of evaluations and benchmarking these models is incredibly nent, right? It's only been around as long as LLMs have been around in the last couple of years. And because of that it's a kind of like a fractured field, right? There's no there's no standard playing field for how these labs uh report um data on on on benchmarking. You know, some may emphasize with Grock 4 recently, we saw huge amount of emphasis on humanity's last last exam, right? And and less so on other benchmarks and some models will come out without any benchmarking data at all. Right? um and without some kind of essentially there's a lot of heterogeneity in how these labs report the results and what it what it leads to for me is a situation where I think we're at risk of struggling to actually compare the models on any even playing field but there's of course bigger

3:15 questions as well about you know models are often lorded for doing the highest score in humanities last exam so we know from a technical perspective the model has advanced above its rivals but for me and I guess my kind of core or argument in this space is if you just rely on those technical metrics, you miss half the point, right? Like these models are designed for humans to use at the end of the day. Most of the users are humans and simple performance on these exams doesn't necessarily correlate to a good user experience. All the frontier labs really uh need to start kind of having uh human preference leaderboards a little more front of mind alongside all the technical metrics. People are increasingly using these models for very sensitive topics and questions for mental health for uh how should they should navigate problems in their lives and there is no oversight on that and in any other area where these topics are

4:09 discussed there is a lot of regulation and then a lot of kind of ethical conduct built into it. Whereas here is kind of the wild west at the moment and some companies are taking it more seriously than others and trying to study the ways in which humans are uh using the models for for more personal topics and and problems and we've seen some pretty starking examples recently with with Gro 3 and Mecca Hetler and uh it does raise questions about how how thin of a veneer is the safety training on top of some of these models.

4:39 Well, there is no leaderboard for safety, right? like there's no metric like we we don't grade LLMs by how safe they are. In fact, it's not really even in the question apart from some researchers. So, I mean I I would argue that that should be just as important as how fast or smart the model is. You know, how safe is it for the people to use? There's been a lot of interesting research coming from anthropic in that direction with regards to safety with regards to alignments of the models and using constitutional AI and various approaches that they've um that they've explored and also around mechanistic interpretability just peering kind of behind the curtains of the models and understanding how an input produces a certain output which kind of features concepts which circuits get activated along the way and kind of tracing the thoughts essentially of these models and trying to isolate where potential

5:35 problems may emerge. So work of this kind is very important and raising the confidence that these models will be able to handle novel situations in safe ways. I think the the one other thing I'd say is um given that chatbot Arena is the number one and frankly pretty much the only human preference leaderboard out there for for LLMs, it's really important that we actually understand what's going on behind the scenes. So obviously, you know, Chat Bolerina is entirely open source. People go in, they put in a prompt, they get response from two different models. They then say which one is better. And that paper found that actually what was happening in the background is that some companies are getting access to a lot more private testing uh in the background than others. For instance, before Llama 4 launched, we saw Meta released 27 models on the arena. But of course, only one was actually reported

6:23 in the end, which obviously undermines the integrity of the arena because the more comparisons you have for your model, the more access to prompts you have, the more data you have to refine a better model that's better at the arena. and it adds an element of bias into into the data which is very very hard to get around. There are other issues that were called out and issues that we've seen ourselves which we think kind of dictates the need for a more rigorous and and methodologically sound approach to doing these kind of human preference data sets amongst us. I think we had a pretty good idea um beyond beyond the criticisms in the leaderboard illusion paper. We think that it actually that paper didn't really touch on some of the other things we we think should be cared about when you're doing human preference evaluation. I think for me there's kind of three big areas I think where where

7:07 we've sought to improve. First of all, as you mentioned sample. So obviously the sample for uh the chatbot arena is anybody right. We don't know anything about them. We don't collect any demographic data. Uh so they are just people going there anonymously prompting the models and giving their preference data. Now obviously that's great. You get a huge amount of data which is fantastic but you know nothing about the people giving the data which is fairly suboptimal. Um then in terms of specificity any for anybody that's used chat arena all you're doing is saying I like this response more or I like this response more right in the real world that kind of data is is useless in a sense right it it gives you a really nice way to make a nice leaderboard of AI models but it tells the companies nothing about why that preference um has been has been uh given. So in in our approach we sought to um mitigate that

7:58 by actually splitting preference down into its constituent parts. So things like how helpful do people find models? What's the communication like? How adaptive do they find it? What do they think of the model's personality? And that when you get ratings on those kind of factors, what you actually get is an actionable set of results that say, okay, your model is struggling with trust or your model is struggling with personality. That's where you need to be focusing to really actually build a model that is good for real users in the real world. But there's no QA in the sense that okay, I could go in and I could just say hello or I could say absolutely nothing or I could have a multi-turn conversation and completely wander from you know how big is the sun to how long is a snake you know like just uh topic wandering which I don't think is a really good nuance view of models. So we've built in to our

8:42 structure where participants come in and they have multi-step uh conversations with models. we built in QA that actually says, you know, if if you put low low effort into your question or you start wondering, we're going to penalize you and three three of those and you're out. So, those are the kind of principles I guess we built the leaderboard around.

9:02 I would just touch upon uh the methodology that we've used, which is true skill. It's um a framework uh if you will developed by Microsoft for estimating the skill levels of players on Xbox Live. So they take into account things like randomness in games, um, um, changing skill levels across time, whether someone is having kind of a fluky win streak versus a seasoned player that consistently performs well. So all of these things that we thought would be good to take into account. And uh it's a very flexible system that estimates probabilities with basing distributions with kind of a mean and a variance that gets narrow and narrow over time as the system kind of learns about the outcome of these battles or these comparisons. Most importantly, it's based on information gain. So the way the way we pick the next pair that should occur in the tournament is based on how much we will learn from these

9:58 models going head-to-head. How much are how much information are they giving us? how much are they reducing the uncertainty and we kind of order the queue of pairs according to that and that gets us to a place of minimized uncertainty as fast as possible as fast as we can. It's a really flexible approach. Uh we can run separate tournaments like we've done with our demographic groups. We have around 20 demographic groups and we've run separate tournaments for them and we can consolidate the findings for each tournament to obtain kind of a an overall leaderboards that is much less uncertain than any of the individual tournaments or leaderboards that we can um produce from any of the demographic groups. So it it really allows us to slice and dice data in any way we want and we can easily add more demographic groups, more models over time. uh we're yeah developing in in in the open and welcoming feedback.

10:51 I think one of the things they pointed out in the leaderboards illusion paper was that actually some models are sampled considerably higher than other models and the I I believe that the the folks behind chatbot arena said that's because people come to the arena to play with the latest models, right? Which is all well and good. People want to play with with the state-of-the-art, right? Which is great, but it doesn't lead to an efficient sampling method. It it essentially means that some models get a lot more battles than others. They therefore get a lot more data. They therefore get a lot better in the arena.

11:20 And there's a pretty strong relationship between the number of battles and the place on the leadboard. We only ever do battles based on the need from the data. So the uncertainty is high for a specific model against another specific model. We conduct a battle for that to lower that uncertainty. So it's all driven by the data. It's very computationally sound because we don't actually make any more comparisons than we need to. Um, and it allows us to really get to a point where models are strongly differentiated based on uncertainty.

11:48 If we have a certain goal with regards to uncertainty, in order to fully differentiate the models at the confidence interval that we're interested in, we can just conduct more battles until we get there. The control is in our hands. Uh, in essence, we just need to recruit more participants in order to get to that level of certainty. The way we've sampled for this study, we're obviously using uh our own participants from the prolific platform, but we've sampled effectively in based on the census data that we have for both the US and the UK. So, the long-term vision of this would obviously be a more global product, but at the moment, what we do is we we stratify our our sample i.e. our participants who are giving us this feedback by demographics like their uh their age, their ethnicity, their political alignment. And we have an awful lot of data from from censuses that tells us, you know, each country is

12:36 made up of this certain proportion of these demographics, which essentially allows us to say that when we've amalgamated all these findings and we find that leading model, we can very confidently say that that model is preferred by as representative a set of the general public as we can possibly get. So hopefully in that sense it's a lot more related to like the the real world preferences of people in the world rather than a very potentially skewed and biased subset that might be uh responding to the chatbot arena because we we ran our first one as an MVP a proof of concept that was a lot more about kind of um proving that we can do this in a rigorous and methodologically sound way. Uh when we actually ran that we only ran it with 500 participants. It gave us a lot of insights about how we build humane which is our leaderboard that we're working on at the moment. Now that leaderboard is actually running as

13:28 we speak in the background. We're still having battles. So we expect to be able to have more uh data from that. But what I can say about the the first uh the first round that we did models tended to perform across the board of the six models we tested which were leading models at the time they performed a lot worse on personality metrics and background and culture metrics as opposed to things like helpfulness, communication, adaptiveness. What that really signals is that there's there's some some uh I guess more subjective aspect of these models which people are less impressed by potentially just they were doing tasks that don't elicit a personality in the model or they don't elicit the model talking about background and culture. Also the model doesn't know their background and culture so it's very hard to align with them. But the other uh possibility is that potentially models are just not

14:15 very good at that. And that would be uh potentially an effect of the data they've been trained on. Right? Because we know very little. I mean obviously models are trained on the entire internet. But it when you train a model on the entire internet, do you get a personality that really represents what people want? Right? And and from this testing, we found that generally people were less impressed with model personality or its ability to have an understanding of their background or culture than with more kind of I guess objective measures.

14:46 And obviously a lot of um these models have undergone extensive fine-tuning to tailor their personalities or tailor how they approach answering questions that are different across the different companies. But uh we've observed recently that there has been an increase in psychop fancy or this kind of peopleleasing behavior of models and people generally don't seem to like it. One thing that this the results of this experiments or and the later data set will allow us to answer is what is a correlation between telltale signs of psychopansy and a down votes in the personality metric that um uh does that uh influence people's decisions on which model they prefer? uh we can um perform various types of post-processing and analysis of the data to identify the levels of psychopancy that's observed in the data sets and to try to identify interesting relationships between the feedback that people gave kind of uh

15:44 more uh model driven kind of LLM as a judge uh oriented uh analysis of of the conversations and kind of classification of various patterns and and and what the models exhibit. So, it's quite quite interesting to see what we'll find.

Questions about this episode

Who speaks in "Why High Benchmark Scores Don’t Mean Better AI [SPONSORED]"?

Andrew Gordon (Guest), Nora Petrova (Guest).

What is discussed in "Why High Benchmark Scores Don’t Mean Better AI [SPONSORED]"?

In order: 0:00 Prolific, Andrew Gordon, Nora Petrova, Grock 4; 3:00 Chatbot Arena, Anthropic; 6:00 Chatbot Arena, Llama 4; 9:00 Chatbot Arena; 12:00 Prolific, Chatbot Arena.

What is mentioned in this Machine Learning Street Talk episode?

7 people, works and ideas are mentioned, including Prolific, Andrew Gordon, Nora Petrova, Chatbot Arena, Anthropic, Grock 4. Each one on this page links to its own page with every episode that mentions it.

How long is "Why High Benchmark Scores Don’t Mean Better AI [SPONSORED]"?

The episode runs 16 min; the transcript is about 3,124 words in 23 timestamped sections.

More from Machine Learning Street Talk

• All analyzed episodes
• Books mentioned on Machine Learning Street Talk
• People mentioned on Machine Learning Street Talk
• Topics mentioned on Machine Learning Street Talk