Transcript

AgentX-AgentBeats Competition Info Session

From Berkeley RDI Center on Decentralization & AI · Episode page · Watch on YouTube

· 1 h 17 min · 9,271 words · 95 timestamped sections

This is the full, timestamped transcript of "AgentX-AgentBeats Competition Info Session", an episode of Berkeley RDI Center on Decentralization & AI published November 7, 2025 (1 h 17 min, about 9,271 words). It mentions 8 people, works and ideas, including AgentBeats, Dawn Song, Xiaoyuan Liu, Berkeley Center for Responsible Decentralized Intelligence, Agentic AI MOOC.

Who speaks in this episode?

What is discussed, and when

  1. – 10:00: Agentic AI MOOC, AgentBeats, Dawn Song, Berkeley Center for Responsible Decentralized Intelligence
  2. – 20:00: AgentBeats, Dawn Song, Xiaoyuan Liu, Agentic AI MOOC
  3. – 30:00: AgentBeats, Xiaoyuan Liu, tau2-bench
  4. – 40:00: AgentBeats, tau-bench
  5. – 50:00: AgentBeats, Dawn Song
  6. – 1:00:00: CyberGym, tau-bench, tau2-bench
  7. – 1:10:00: AgentBeats, Dawn Song, Xiaoyuan Liu, tau-bench
  8. – 1:17:02: AgentBeats

Notable quotes from "AgentX-AgentBeats Competition Info Session"

  • I'm a professor in computer science at UC Berkeley and also a co-Director of a campus-wide center called Berkeley Center for Responsible Decentralized Intelligence RDI.
    about Berkeley Center for Responsible Decentralized Intelligence · at
  • And we hope that the community and the participants can join the competition to both actually help together build public goods for agent evaluation and also be a great learning experience for everyone.
    about AgentBeats · at
  • So I guess for the next part, I'll be walking through some of the implementation details on how to use AgentBeats or integrated AgentBeats for an agent evaluation.
    about Xiaoyuan Liu · at
  • So imagine that-- so first of all, the tau-bench is-- just for the context, the tau-bench is a tool use benchmark that the assessor will have a bunch of tools, and the assessee…
    about tau-bench · at
  • So CyberGym is actually our recent work as a benchmark to evaluate AI capabilities in cybersecurity, in particular for evaluating agents capabilities in identifying…
    about CyberGym · at
  • And I also wanted to quickly mention the tau2-bench, which is a next generation from tau-bench, which includes more dynamic interaction between users and the agents.
    about tau2-bench · at

Who and what is mentioned in this episode?

People

  • Dawn Song — Person · Main speaker · 4 mentions · first at
    Leads the info session, motivates the AgentX–AgentBeats competition, and explains the AAA paradigm and judging criteria for evaluation benchmarks.
    And so we welcome everyone and the community to join this AgentX-AgentBeats competition in conjunction with the Agentic AI MOOC.
  • Xiaoyuan Liu — Person · Co-presenter / technical lead speaker · 3 mentions · first at
    Introduces the AgentBeats platform implementation details, explains how to agentify benchmarks and integrate agents via A2A/MCP and the AgentBeats controller.

Companies & orgs

Products & tech

  • AgentBeats — Product · Business focus · 19 mentions · first at
    A platform for open, standardized, and reproducible agent evaluation and risk assessment.
  • Agentic AI MOOC — Product · Business focus · 4 mentions · first at
    A Massive Open Online Course in conjunction with the AgentBeats competition.
    And so we welcome everyone and the community to join this AgentX-AgentBeats competition in conjunction with the Agentic AI MOOC.
  • tau-bench — Product · Case study · 7 mentions · first at
    A tool use benchmark for computer use agents, mentioned as an example of a good evaluation system.
  • CyberGym — Product · Case study · 6 mentions · first at
    A benchmark to evaluate AI capabilities in cybersecurity, mentioned as an example of a good evaluation system.
  • tau2-bench — Product · Case study · 3 mentions · first at
    A next generation from tau-bench, includes more dynamic interaction between users and the agents.

Full transcript with timestamps

DAWN SONG: Well, great. Hi everyone. Thanks a lot for joining this info session for AgentX-AgentBeats competition in conjunction with the Agentic AI MOOC, the Massive Open Online Course. Thanks, everyone, for joining. And yes, my name is Dawn Song. I'm a professor in computer science at UC Berkeley and also a co-Director of a campus-wide center called Berkeley Center for Responsible Decentralized Intelligence RDI. And RDI is hosting both the Agentic AI MOOC and also this AgentBeats competition. So first, I will give some quick overview of why we are doing this competition.

And then we'll go into more details about what you need to do for the competition and more information and so on. So first, the competition focuses on agent evaluation. So first, what is evaluation for AI models and agents? So these evaluations are essentially systematic, repeatable measurements built using these benchmarks that provides a structured way to measure different aspects for models and agents. And this includes different types of capabilities, as well as risk assessment as well. So why are we doing a competition on agent evaluation?

Because actually agent evaluation is really important. Agent evaluation is important for a number of reasons. It provides important measurements across models and agents for both capabilities and risk assessment. And it guides safe and effective development, as well as deployment decisions, and helps expose both weaknesses and strengths and help compare different models and agents as well to help make the best decision in terms of how to both improve models and agents, as well as how to choose the best model and agents to deploy for real-world applications as well.

And hence, reliable and high quality evaluation of models and agents is critical develop and deploy effective and safe agents in real world applications. And also a really important part about evaluation is that evaluations essentially help set the goalposts for the community for developing better AI models and agents. There's a saying, you can only improve what you can measure. And essentially AI's revolution is upper bounded by evaluation. And also we have seen, essentially, the history of AI development essentially has been marked by the different types of evaluations and so on.

So one thing I want to mention here is, so far, actually, when you look at the community, most of the evaluation has been focused on the model evaluation. However, as we develop agents, evaluation that just focus on the model level is insufficient. So essentially, we need to transition from model evaluation to agent evaluation as we now move to develop agents and also to perform evaluations for agents. So there are a number of key differences between model evaluations and agent evaluations. So for models, for example, these large language models, they are static and they can be-- right.

They are oftentimes just text-to-text, and they can be multi-modal as well. But in general, these model evaluations focuses on just providing prompt input and evaluates the outputs of the model. And it's relatively-- in this case, it's relatively straightforward to do the evaluation given the benchmark. But however, when we look at agent evaluation, it's much more complex. As we have discussed in the Agentic AI MOOC, agents, the overall agentic AI system can be very complex. They do use the model as a key component to-- and then they use the model to do, there's planning to use and reasoning and so on.

And also agentic AI systems have memory storage as well. And agents operate in dynamic environments. And all these factors essentially require more complex evaluations when we do agent evaluation than just the simple model evaluation. So now, since as I mentioned, most previous approaches actually focus on model evaluation, and as agent benchmarks have been developed, actually most of the existing agent benchmarks still carry the same paradigm that they are-- actually these evaluations are LLM-centric design and with fixed harnesses.

So essentially, most of today's agent benchmarks, I would say they have the wrong separation points, where they actually combine the benchmark with fixed harnesses and only expose the model, the interface between the model and the agent benchmark. And hence, in this case, one can easily evaluate, with the same fixed harness, the benchmark evaluation across different models. But it's actually very difficult when you want to evaluate different agents with different harnesses. And hence in this case, when we want to evaluate a new agent on today's existing agent benchmarks, there's a high integration overhead.

One has to figure out for each new agent how to do the integration with the current agent benchmarks. And also this causes an issue with the test production mismatch, where one has to often change the production agent to adapt to the testing benchmark. So you actually may not be able to actually evaluate the production agents as is. So these are some examples of the key limitations of existing agent benchmarks. As you can see, in order to evaluate agents with different benchmarks, you would-- if you have M benchmarks, oftentimes you have to do M integration adaptation to integrate and adapt these agents to the M different benchmarks.

And if you have N different agents, essentially have this multiplication N by M, the integration efforts. So this motivated us to develop a new paradigm that we call agentified agent assessments, AAA. And this serves as a new paradigm for open, reproducible, and standardized agent evaluation. And in particular, the key point of a agentified agent assessments paradigm is that we enable what we call identified evaluation, where the benchmark evaluation itself, the assessor, is actually an agent itself. And this assessor agents will then directly communicate with the assessee agent, the agents that actually need to be evaluated, through standardized protocols such as HOA.

And also we use MCP as standard protocol for two use in this new paradigm as well. So using this approach, we enable standardization and utilizing the existing standards such as HOA and MCP. And also we enable reproducibility by utilizing an assessment control protocol that controls for the setup and the environments for the agent evaluation. And in this case, one way to think about it as this new agentified agent assessment paradigm enables the key separation and the right interface separation between the evaluation benchmark and the agents.

So with this approach, it's very easy to bring your existing agents and then to just be evaluated elevated by the accessory agent. That will actually call green agent in the competition and the accessory agents the currency we call the purple agents in the agent beats competition. So again, this enables assessee agent to easily be evaluated and also this enables different harnesses and different models easily to be evaluated. And when compared to traditional agent benchmarking, this agentified agent assessment paradigm has a number of advantages.

In the traditional agent, benchmarking is again primarily focused on LLMs with fixed harnesses, whereas with the AAA paradigm, any agent conforming to the A2A protocol can be evaluated by the assessor agent. And in the traditional agent benchmarking, it provides benchmark-specific and implementation-dependent interface, whereas in the AAA paradigm it provides standardized A2A interface for task management and MCP for two use and to access. And in the traditional agent benchmarking, it's prone to test production mismatch, as I mentioned earlier, and it's mainly used for reference, whereas in the AAA paradigm, it directly reflects production level performance using this evaluation paradigm.

And for multi-agent assessments, in the traditional agent benchmarking, it's actually very difficult and requires bespoke integrations, whereas the AAA methods it natively supports multi-agent assessment through standardized interfaces and platform-level coordination. So it's actually provided and enabled natively.

And so these are the advantages and the motivation for providing this new paradigm of agentified agent assessment. And in addition to this paradigm and the issues addressed by this new paradigm, there are also additional obstacles for building impactful agent evaluation. As the overall system implementation can be complex, requiring integrating of multiple LLMs, navigate diverse agent frameworks, manage observability, environment setup, and infrastructure for agent deployment, monitoring, and leaderboard management.

And also overall, there is a lack of openness and there's a lot of fragmentation. Currently, there's no unified platform that can transform research prototypes into widely accessible reusable evaluations. So this essentially motivates our new open platform AgentBeats for open, standardized, and reproducible agent evaluation risk assessment, which Xiaoyuan will discuss more details. XIAOYUAN LIU: Sure. So here, I'm very happy to introduce you our current ongoing effort for building this AgentBeats platform. Essentially, it's a platform compliant to the AAA standard that we just mentioned.

And the goal is to build, again, the standardized, open, and also reproducible platform that satisfies, like our previous assessment-related requirement. And for standardization, in addition to using A2A and MCP, we also provide a unified SDK as long as a consistent workflow for managing agent assessment and also agent instantiation. Also targeting openness, we plan to use the platform for publishing like public agent maintaining benchmark, and also provide a set of hosted environment for open evaluation. In terms of reproducibility, we support a new control protocol that allows the auto reset of the environment and also allow the hosted runs of agents.

And while the assessment is ongoing, we also enables multi-level trace logging for better observability. And the platform is easy to use. Essentially, you can use one file to instantiate a new agent, and also there will be both on-platform and self-hosted options just in case someone wants to host private competitions. And to just make the platform more extensible, we also support rich integrations of different types of agent as well as different types of scenarios. And below you can see a general workflow for how AgentBeats SDK, as well as the AgentBeats platform service can simplify the overall development and evaluation of agentic systems.

And the platform is targeting the use case for evaluating the agents and also letting different agents to compete on the comparable tasks. It also encourage contributing to new environment and benchmarks, allow people to collaborate using agents developed by others, and also let the agent improve. We support mainly two types of evaluation mode. One is the benchmarking mode for single agent, is best for having a unified scoring and leaderboard for a single agent task. We also have an arena mode, which basically means that you might have multiple agents joining the same assessment, and based on the type of assessment, either is like a one on one or it's more of a multi-agent sort of team fight self-evaluation.

There might be different ways of assigning a score. And this is best for adversarial multi-agent evaluation or collaborative competitions. And just as a general concept walkthrough, any A2A compatible agents would be also compatible to the AgentBeats platform as long as they support the basic task fulfilling to using an optionally memory and other component. There are, in general, two types of agent-- the assessor agent and the assessee agent as introduced earlier. And just for a quick example, imagine that you have a chess game between GPT-4o agent and also GPT-5 agent.

Then these two agents will be your assessee agent while the agent who actually judge the board will be your assessor agent. And we organize the evaluation in the format of assessment. Each assessment would be a multi-agent procedure between one of the assessor agent and the other agents. Each assessment will reflect one or more matrix of all the participating or assessee agents, reflecting on their score chain on the leaderboard, and also the assessor agent is responsible for reporting the final result to the platform in the end.

So the platform itself provide a basic feature of agent registry for people to discover new agents. It also provide the agent controller for the state task state management, as will be also introduced in later. We also provide a feature for assessment kickoff to orchestrate the collaboration procedure between different agents for getting the assessment result. And there are also extended features and a way of using it for assessment tracing, leaderboard, proxying the MCP calls for access control, agent hosting, et cetera, et cetera.

And we'll be releasing more details in the future blogs. DAWN SONG: And yeah, thanks, Xian. And so we welcome everyone and the community to join this AgentX-AgentBeats competition in conjunction with the Agentic AI MOOC. And we are very honored and grateful to have the sponsors provide-- we have over $1 million in prizes and resources, thanks to the different sponsors here. And we hope that the community and the participants can join the competition to both actually help together build public goods for agent evaluation and also be a great learning experience for everyone.

So now let's talk a little bit about the detailed logistics for the AgentBeats competition. So for the competition, there will be two phases. The first phase that is going on currently is to focus on building the accessory agents, also called the green agents here. So in particular, for phase one, the participants will be building these accessory-- green agents that actually define the assessments and automate the scoring. And in particular here we have several different tracks and different contribution types. So first, let's talk about the contribution type.

So one type is to actually port or agentify and extend an existing benchmark. So the goal here is to transform an existing benchmark into a green agent that actually runs end to end on AgentBeats. And on the AgentBeats competition website, here, you can also see more examples of benchmark ideas that actually list out a number of example existing benchmarks that you can go agentify and extend. So that's one contribution type. Another contribution type is to create a new benchmark. In this case is to design a brand-new assessment as a green agent with novel tasks, environments, automation, and scoring.

And we in particular welcome the participants to develop new assessments in diverse domains that have sufficient difficulty levels and also are realistic. In particular, for example, you can take your model after your even daily workload and so on, and develop these new assessments as green agents to evaluate the agent capabilities, in particular in domains that you are familiar with. And also we have a custom track which we'll talk about in a minute. So these are different contribution types. And then we also welcome for each of these types, either for extending agentifying existing benchmarks or to create new benchmark, for each of these contribution types, we welcome contributions across different very diverse application domains.

So these include coding domains, web agents, computer use agents, research agents, software testing, and game agents. And in, for example, Web3, decentralized finance, cybersecurity agents, healthcare, finance, legal domain, and in general, for agent safety and agent security and multi-agent evaluation, and also many other domains, which I won't be able to exhaustively list here. So again, we welcome the participants to consider these different domains. So pick a domain that you are interested in. And also ideally, you are familiar with.

And so the registration phase has started. So for all the participants, you need to register individually as a participant. And also the team sign-up has opened as well. So we welcome you to form teams and do the team sign-up as well. So that's phase 1 for developing the green agent, which is the accessory agent. And this will run until end of this year. And then after this one, we'll enter phase 2 in early spring next year. And phase 2 focuses on developing the purple agent, which is the assessee agent. So in particular, after phase 1, we'll select the top green agents from phase 1 and have participants to compete on the public leaderboard for the selected top green agents from phase 1.

And can you go back? So I'll also talk about the custom tracks. So in particular here, currently we have two main custom tracks. So in general, as I mentioned in the previous slide, you're welcome to develop green agents in different domains. And in particular, thanks to the sponsors, we also have particularly two custom tracks if you are interested in these two specific areas. So one is agent security developing essentially benchmark evaluation for red-teaming and automated security testing. And this track is sponsored by lambda.

And more details will be announced soon. And the second current custom track is to extend the tau2-bench. And this is sponsored by Sierra. And you can again go to the website to look at the detailed information about how tau2-bench and also more details about this custom track will be announced soon as well. Please stay tuned.

And thanks to our sponsors. There's a very rich resources and also prices provided by the sponsors to the participants. So for every participant-- you can look at the resources available here. So essentially every participant actually just by participating in the competition, you can get close to $500 at least worth of credits for just participating in the competition. And we also have various prizes for the winners of the competition provided by the different sponsors. And more resources and prizes will also be announced later as well.

And also please check the competition website for more details on the key dates. Yes, we hope everyone have a great time enjoying first for the phase 1, building the green agents, the assessor agent. And also we have the Discord. We welcome everyone to join the Discord. So if you have more questions about the competition and AgentBeats and so on, please post your questions in the Discord as well. XIAOYUAN LIU: Cool. So I guess for the next part, I'll be walking through some of the implementation details on how to use AgentBeats or integrated AgentBeats for an agent evaluation.

This have two parts. The first part I'll be showing how, in general, we can agentify existing benchmark by trying to customize its interface and standardizing it towards A2A. And for the second part, I'll be talking about how for a standardized A2A or agentified assessment, how we can integrate it with AgentBeats to make it object that can be used in a general agent assessment in a collaborative way. So first, there are in general three steps for agentifying existing benchmark-- oh, by the way, feel free to scan the QR code here or go to our official website to read more about in general this example.

So the first step for agentifying an assessment is to try to sort out the interface. And there are two general principles that can be applied here. The first one, in general, just by the definition of the task, humans should be able to also solve whatever is presented to the agent and with the given information. And under the assumption that it's human-solvable, in general, it's recommended to make the solving procedure as agent-friendly as possible just so that the agent can solve it easily and, in general, achieve more reasonable score.

Imagine that if you can make a question to be a multiple-choice problem, do not make it a slot-filling or a text-generation problem. And that might, in general, make the assessment more compatible. And a general example for how to exam this or how to practice these principles include, for example, in a web browsing agent, you might have the option to provide your assessee agent the URLs to access those web page, or you can follow the same paradigm as say, the current days, like in browser jean, you provide a bunch of tools for navigating through the web page and get the standardized response.

And in this case, since in general human would prefer the URL version, since you can just open it with your customized browser and use your own sort of way of operating on a website, then it's actually recommended to organize it so that your web browsing task is actually providing a URL, rather than just translating things into two actions. And similarly, for coding agents, there's a difference of providing the coding environment versus if you just provide a repo to be changed and expect the agent to write you the patches.

And in general, if you compare it to the human-related assessment, just imagine the first scenario to be you share your laptop with, let's say like given assessee versus you just provide a public GitHub repo link and then just ask others to do PR. In this case, I think too, if you want to examine the agents ability to interact with, say, the code editor, then the first way is preferred, otherwise the second way might be more cleanly handled. And a recommended way to sort out the interface would try to read the original paper if you are integrating-- if you are migrating from an existing benchmark, or just think about a task formulation for the new types of benchmark.

And in general, it's also recommended to read the code base if you are migrating from an existing benchmark and think about how to actually deliver the same piece of information but within the A2A format with minimum code intrusion as possible. So here, we actually provide an example of using tau-bench. And as you can probably see, the figure from the tau-bench paper actually clearly present what would be, in general, interaction structure. And if you go into their code base, you will see that there's this very clear interface of environment and agent that you can extract.

And as long as you can separate it from this layer, it's possible to actually make it into a compatible. And specifically for this tau-bench example, there remains like two key challenges for agentifying it and standardizing it with A2A. So the first is that you will have a cross-agent tool use scenario. So imagine that-- so first of all, the tau-bench is-- just for the context, the tau-bench is a tool use benchmark that the assessor will have a bunch of tools, and the assessee will be trying to use these tools to solve, let's say, a given problem.

And in this scenario, since there are two-- since we are agentifying it and the assessor is actually a different agent from the assessee, you should figure out a way to actually pass the two lists from the assessor to the assessee just to make sure that the assessee can respond in the expected format. And in the original repo, the tool is directly provided to the chat completion interface. It actually doesn't need any sort of separation. But since now we want to make this compatible and standardized so that everyone can run this evaluation, we need to think about how we can pass these two lists from the assessor to assessee.

And there are, in general, three ways. The first way being the simplest, that you just expect the participating assessee agent to have the knowledge about these tools and then just operate based on these given formats. However, this actually defeats our purpose of agentifying assessment and also is less standardized. And the second way is to try to ask the assessor to explain the tool-access requests to the white agent, and then ask for what tool names that the assessor want to call. So while this is achievable and also this is the way we use in our example, one of the potential problem is that this cannot leverage the agent internal tool-call mechanism.

Say if you have hyperplanning or organization or some sort of harness regarding the tool selection, then this part cannot be properly evaluated. There's also a third way of doing this, which is to ask the green agent or the assessor agent to provide MCP server for accessing the tool, and then just get the connection or get the URL to the assessee agent. And while this being the most general approach, there will be some coding complexity. And also this requires the dynamic discovery ability of the assessee agent. So there are trade-offs.

And also the second challenge is that when you migrate the evaluation, the tool trace might not be directly visible to your assessor agent. Thus, a gradient can be problematic. And in general, to just make sure that you have rather independent agentified assessment, there can be three parts. So first, you want to build a green agent or the assessor agent and the assessee agent or white or donor as the different participant in this multi-agent procedure. And also you would like a kick-off script which send a message to the assessor agent to kick off the whole procedure.

And later within the AgentBeats platform, this can be simplified by having a platform-owned component to send this message. But just for a complete example, you might want to have this separate component. So implementation-wise, you would like a kick-off script that looks like the following. So you will try to send out this kickoff message to the green agent, and that's it. And then for implementing the green agent, what you need to essentially do is to implement an agent interface. And in this case, a coding based workflow would actually solve the problem that you just load the corresponding environment and then issue and simulate the conversation with the assessee agent.

And it might be slightly different for MCP-based implementations since you need to set up the separate MCP server, but the general workflow remains the same. And similarly, you will have a participant agent that can be very simple, as simple as just a general agent since they are the candidate to be tested. And again here, the purpose is that for any agent under any agent harness, you can incorporate it in a standardized test. So once you integrate, once you have made the agentified assessment and standardized your agent interface, the next step is to leverage the AgentBeats platform or the AgentBeats format to make it reproducible.

And there are also some potentially helpful materials that may help you with this overall development procedure. So for the second part regarding how to integrate your agent with the AgentBeats, so now you have the agentified assessment and also having the baseline agent and the launcher, the integration just takes three steps. So the first step is to wrap your agent with what we call the AgentBeats controller, which will be also discussed a little bit later. And the second is to deploy your agent to the cloud, to some sort of cloud platform, to obtain a public IP as well as protected by TLS.

And at the last step is to connect it to the AgentBeats platform so that others can actually leverage it for a multi-agent assessment. So for AgentBeats controller, it's actually a lightweight component that can help you manage your agent instance. You have several key responsibilities. The first, it actually exposes a service API for managing your agent state. For example, for tracking agent liveness, restarting the agent, or checking the agent log. It also detects and start or restart your agent based on the API request.

And additionally, just for the purpose of make it friendly to a microservice architecture, you also have a feature to proxy all the requests that they send to the internal agent instance. And just for human-friendly, UI also provide a management panel for both debugging and also checking the internal status. And the reason you need such a component is that other users can actually test your agent without manually restart-- manually restart your agent between runs, or just reach out to you to say your agent need some state reset.

So the procedure is also simple. You install the package. You set up an executable named run.sh, which in general tells the system how you would start your agent and also make that script executable. And then you run the controller from the package. And what you get is that you will have, first, a local management page for monitoring the agent status. You will also have a proxy URL for accessing your agent, similar to the one that is starting in local host. And also you will have the-- you can test whether this is set up by just checking the agent card.

And here is the example for what you will see if you are using this controller. So essentially, you will try to manage your agent instance internally. And there's also ongoing like scaling feature that might help you to manage multiple agent instances in the future. And you can toggle Reset just from the controller API. So once you have this controller-- so once you have this controller, you might consider deploy your agent with a public IP and TLS. And there are also, in general, a few steps for deployment, including provision of VM, also install the program, getting the certificate.

And for a modern alternative, you might want to containerize your agent and potentially with some sort of Buildpacks or Dockerfile and also deploy it for microservice or function calling, sort of like service just for getting that TLS and also autoscaling automatically. So the steps are also quite simple. Here's an example. So for example, you can set up the profile and then just build with Buildpacks and then push it to a Docker registry. And the benefit is that you don't need to manually set up HTTPS anymore. And also the management of your agent will be simpler.

So the last step is to actually publish your agents on the AgentBeats platform. And once your agent is publicly accessible, others can discover it and also including it in general assessment. And the procedure is also simple-- visit the website, fill in the form, and then the key information you need to provide here is the controller URL that you set up earlier with public IP, and for a complete example coding example of what you need to change from an agentified assessment to AgentBeats integrated agent, please feel free to take a look at this example patch.

So once you have this set up, you also have next step of running assessment and viewing the result from the AgentBeats dashboard. And we also have some advanced feature for hosting agent, et cetera, just to simplify the workflow. But in general, that would be the flow for integrating with the AgentBeats for agents. DAWN SONG: OK, great. OK, so yeah, thanks, everyone. So now you've understood the motivation for this new paradigm for agent evaluation, the agentified agent assessment and also understand-- I've seen some examples of how to build the assessor agent, the green agent.

And so now, we'll also talk a little bit about, for the competition, how the submissions will be evaluated. So first, let's look at given that we are talking about agent evaluation, then first let's look at what is a good evaluation system. So one key question is as we develop agent evaluation, how do we make sure that it's high quality? And there are many different aspects to ensure high quality evaluations. And so as I mentioned earlier, so you want the evaluation to have actually to be challenging enough so that you can actually differentiate strong models versus weak models, strong versus weak agents.

And also we want the evaluation to be realistic. Ideally, the evaluation is more modeled after real-world useful tasks and real-world settings and so on. And also the evaluation, we hope that it can actually give a fairly accurate assessment and with no bias or low bias and so on. So there are many different aspects that one actually needs to pay attention to. So in particular here, as one example of important aspects, is that you want to make sure that, as I mentioned, the evaluation, for example, your scoring function is actually high quality.

It gives correct evaluation. So outcome validity is very important for making a good evaluation. And so given that depending on the different tasks, there may be different outputs. And this will require different ways to judge the results. So depending on the tasks that you are developing, then you will need to develop appropriate methods, especially if you are building a new benchmark assessment, you need to develop appropriate judging methods. So this oftentimes depends on the different types of the outputs that you need to evaluate over.

For example, for text outputs, some may do string matching. And there are different things you need to be careful if you are doing, for example, substring matching. And sometimes one may LLM-as-a-judge. However, in general, it's really important to ensure that when LLM is used as a judge, it actually does provide high-- does actually provide high accuracy in the evaluation. And in general, especially for this competition, we do encourage participants to try to provide as much ground truth, rigorous rubrics for the evaluation as possible, instead of using LLM-as-a-judge, which oftentimes does not provide a high enough accuracy, depending on the tasks.

So again, the outcome validity is a very important aspect for evaluating your green agents. So here, this is also another example if you are judging code generation then you can use different methods, including unit testing or end-to-end testing, fuzz testing, and so on. And you can also provide evaluation based on environment state changes for a lot of tasks and environments. This actually can provide a very good method for judging the outcome validity. For example, includes the ground truth, and that actually includes all state the achievable after success and check for relevant and irrelevant states.

In general, so this is, in this case, providing ground truth based on environment state challenges-- state changes can be very effective way of performing high-quality assessment. And here are some examples to consider for judging multi-step reasoning, depending answer matching, quality measures, and so on. And also, as you build these assessor agents or green agents, it's important to pay attention to the different ways how the evaluation could go wrong. In fact, building high-quality evaluation can be very challenging.

There are many different ways things can go wrong. So for example, the data can be noisy or biased. There can be data contamination issues in the sense that maybe somehow the model agents, if you are using some existing tests the model may have already seen and write the tests in training data, then this can cause data contamination. And also with agents, sometimes there can be shortcuts, in which case the evaluation can be gamed. You'll be surprised how smart these agents can be, that can try to take shortcuts to game the evaluation.

And also, as I mentioned, ideally, the assessments benchmark should be challenging enough. And also in particular, you want to build these evaluations that are sufficiently challenging not just for today's models and agents, but also you want the evaluation to have sufficiently long-shelf life so that it can still remain relevant even in the coming near terms and even longer terms as well. And also ideally, these evaluations should focus on high-impact tasks and environments and be realistic and practical. And here, I want to briefly mention a few case studies of some examples of good evaluation systems.

And again, so when you are considering what to build for the assessor agent or the green agent, there are a number of questions to consider. In particular, what is a good benchmark and how to construct it. You need to select the initial goals. What is the goal, and essentially what do you want to evaluate? What tasks-- what are the tasks and what are the environments to run the agent to achieve the goal, and how to build the data collection pipeline, how to evaluate the assessee agent? And again, you want to prioritize real-world realistic tasks, have sufficient difficulty levels and ideally have different difficulty levels, and not easy to get contaminated or saturated.

Here are some examples that I just want to briefly mention. So CyberGym is actually our recent work as a benchmark to evaluate AI capabilities in cybersecurity, in particular for evaluating agents capabilities in identifying vulnerabilities, both previously known vulnerabilities, as well as previously unknown vulnerabilities, and also generating what's called POC proof-of-concepts. These are inputs that actually trigger vulnerability. And CyberGym actually was started with a goal to be realistic, and in particular, actually it contains close to 200 large-scale, widely distributed, real-world open source software that contains over 1,500 previously known vulnerabilities.

And CyberGym is constructed using well-constructed environments that are containerized sandboxes to run these programs. And CyberGym actually as a benchmark has demonstrated that the AI capabilities in cybersecurity has been improving really fast and also has been included in Anthropic's system cards for its latest model release, Sonnet 4.5, for evaluation of AI capabilities in cybersecurity. And you can find out more information about CyberGym and how it's constructed on the CyberGym website and in the paper. And as an example, in the paper, it actually performs data contamination analysis to show that there is minimum data contamination issues that needs to be concerned in this benchmark.

And also CyberGym construction focuses on high-quality evaluation that actually has high-quality ground truth built in into the evaluation through the use of runtime sanitizers, which actually gives ground truth and results to show that whether the construct is input is an actual POC that triggers vulnerability, and also separate as, whether it triggers a previously known vulnerability versus a new vulnerability. So that's a CyberGym. Here's another example for tau-bench, which actually Xiaoyuan just did a walk-through in terms of how to agentify tau-bench as assessor agent.

And so again, you are welcome to look at the details of tau-bench in the tau-bench paper, as well as the open source benchmark. As an example, tau-bench provides a very good example benchmark for computer use agents that actually covers a very diverse different types of environments for different types of tasks. And again, you can-- given interest of time, we won't go through the details. You are welcome to go into detail to look at the construction of tau-bench and how it is evaluated, where it actually provides different metrics for evaluation as well.

And I also wanted to quickly mention the tau2-bench, which is a next generation from tau-bench, which includes more dynamic interaction between users and the agents. And also, as I mentioned earlier, we have a custom track for tau2-bench. And again, you can look at the tau2-bench paper and the open source repo for more information.

And then there are some other examples that you can take a look as well, such as the GDPval and also CRMArena. So these are all very interesting examples to take a look. And then I want to just quickly conclude by talking about then what are the judging criteria that you need to consider as you build these two different types of assessor agents or green agents. As I mentioned, there are two contribution types. One is to integrate and agentify an existing benchmark. The other one is to build a new benchmark. So the first one is to integrate and agentify existing benchmark.

On the competition website, there are links, again, as we showed earlier, that you can look at example ideas for existing benchmarks that you can agentify and extend. And in general, the workflow goes as follows. So you want to identify the existing benchmark that you are interested in extending and agentifying. And then you want to then essentially follow the steps that Xiaoyuan has mentioned through his examples with the tau-bench that you want to then agentify the benchmark and build the accessory agents by following the similar steps as what Xiaoyuan showed.

And also after that, we also strongly recommend you to even though this is an existing benchmark, a lot of the existing benchmarks, they do have quality issues and so on. So we do strongly recommend to you to also investigate and to do your own quality analysis for the existing benchmark that you selected. And when you encounter issues, to then take the next steps to ideally help correct some of these issues and potentially even further expand the existing benchmark as well. So for the benchmark quality analysis, you can do it through several different methods.

You can do it through manual validation to sample and check data correctness, clarity, and difficulty. And you can check the evaluator to confirm for metrics. And also the judges that actually do align with the true task success, and also analyze for bias and contamination and other potential issues and limitations as well. And then once you identify issues in these existing benchmarks, you can try to correct the benchmark if there are errors. And also you can try to expand the benchmark to further improve its quality, size, and diversity, as you see fit.

And in terms of identifying issues and correcting issues. So this is a very good example for you to take a look. So there's the original SWE-bench which provides a very good starting points. But it had a number of issues. And later on, with the SWE-bench verified and later on people corrected and identified issues and corrected them. And so now, in general, SWE-bench verified is actually more widely used now as the evaluation for coding agents. So that's the first step for extending agentifying existing benchmark.

As I mentioned, the second contribution type is to build your own new benchmarks. And in particular, in this case, you want to identify realistic tasks. You want to first identify what task and the environments that you want to evaluate, build your benchmarks in. And ideally, you want to develop tasks that actually reflect useful real-world scenarios. And it's even better if it's a domains that you are familiar with and even tasks from your regular workload, and so on. And for the evaluation, you can develop different methods, either with ground truth, with manual developed ground truth, or you can have some semi-automatic or automatic methods as well.

So now, we are seeing more and more multi-agents being developed as well. So we do encourage you to build multi-agent benchmarks as well. So here's a quick step-by-step checklist for building your assessor agent or green agents. Make sure that you choose the tasks you want to evaluate on, and design the environment that agents are being tested and need to run in. And this includes providing the set of tools that agents can interact with, the actions that agent can take, and also the environment feedback to the agent after each action.

And also then design the metrics that your green agents will evaluate with. For example, in this booking ticket setting, the success rate for booking tickets, how cheap the ticket is, whether the ticket satisfies the user's requirements, and so on. And then you need to design the test cases to evaluate your green agent. You need to think about the different scenarios of your purple agents trying to complete the tasks and design the various test cases to evaluate your green agents that can handle the different test cases, where different purple agents will try to be evaluated by the green agents.

And also I want to mention that, in general, top AI machine learning conferences now have data sets benchmark tracks. So for example, this is the NeurIPS, the top, deep learning conference, top AI conference for year 2025. This will happen in December, and it's called for paper for data sets and benchmarks. And so ICML also has its data sets benchmark track actually with the deadline in January. So if you build your assessor or green agents, it's also can be helpful that you can consider the work later on. You can submit the work to these conferences as well.

And also we do plan to offer a certain example tasks and evaluations. So for people who are interested, you can join as well and potentially even co-author papers as well. So finally, for judging criteria, when you are building your assessor agent or green agent, there are a number of aspects to keep in mind. Xiaoyuan will provide you more details for the judging criteria. But these are, in general, the different aspects that you can keep in mind. So this is an example list of different aspects for you to consider for building new benchmarks for the judging criteria.

For example, goal and novelty is your benchmark, important novel and covers new capability space. And how is the scope and scale. It's a benchmark large and diverse enough to give reliable results. Evaluator quality-- are metrics clear? Is your judge or evaluator high quality and consistent? For validation, did you perform manual checks or spot validation and evaluation outputs from your green agents? And for reliability, do your evaluation scripts and green agents run robustly on AgentBeats? And quality assurance-- any bias or contamination checks that you have done?

And realism-- is the benchmark realistic? For example, with real-world workloads instead of toy or unrealistic settings. And impact-- is the benchmark reusable, well-documented, and presented clearly? And then similarly, there are similar aspects for the judging criteria when you are extending and agentifying an existing benchmark as well. So here, in addition to successfully agentifying an existing benchmark, you also want to analyze for quality issues for the original benchmarks that you extended and try to find whether it has any issues, flaws.

And for faithfulness-- is your implementation reproducing results from the original benchmark, excluding the flaws that you fixed? Quality assurance-- is your implementation correcting the flaws in the original benchmark and expanding the coverage of the original benchmark? And evaluator quality and validation reliability, and these are similar aspects that I mentioned in the previous slide as well. And also impact-- is our implementation reusable, well-documented, and presented clearly? So I think that's all. So thanks, everyone, for joining this presentation.

So now we also open up for some Q&A. And Xiaoyuan will help answer questions as well. XIAOYUAN LIU: So shall I go through I guess some of the questions that is previously-- yeah, I guess I'll probably just start from those. And well, in the meantime, please feel free to send more questions in the chat. So there was this question for, does this mean A2A and MCP need to be a part of agentic flow in order for AAA to work? Yeah. In general, the A2A is used for communicating. Think of it as the interface we incorporate for doing task management.

And instead of just setting up a customized one like yet another standard, we're just using A2A based on this data format and way of managing task. And similarly, for two years since MCP is already sort of a common protocol for everyone, we are just leveraging that for green agent to provide necessary tools for agents to interact. Yeah, so there's also a second question that, where's the assessor is coming from-- agent is coming from? Is it available somewhere? We need to be written along with other agent. Yeah, great question.

So the assessor agent or the green agent is actually the main deliverable for the first phase. That means that-- so first of all, it's equivalent to designing a new-- assessor agent is essentially deciding on new benchmark but also along with the standardized interface and also reproducible sort of ability that-- so yes, it should be coming from, I guess, the submission for-- come as the submission for this competition. And I'm also going to go through some questions from other channel. You will not provide any API for why purple agent to use for testing it with our green agent.

So we are not providing-- well, we are not providing a standardized agent is currently like a most modern agent framework do have their A2A support. And in general, feel free to use those, or like off-the-shelf example as the general green agent to test against your target. In general, a good green agent, a good design like a assessor agent, should be able to be general enough to test any general assessee agent. However, with that being said, we also recommend you to also build a better baseline assessee agent separately just to demonstrate that for certain improved agent, that your benchmark or your assessor agent can distinguish them and actually measure the ability.

For example, if you're building a web browsing environment for a web browsing green agent, then potentially you might want to use or incorporate other like agents with web browsing tools or with web browsing ability, either from a visual version or a text version, or a visual input, or a text input to provide more baselines. So there was also a question about some current, I would say, difficulty in using the website. Well, there will be-- so first of all, there will be a different-- actually a different set of UIs and the upgraded version of the website for the official submission for this competition.

So I would personally recommend to start with the agentified example, or the tau-bench example for agentifying the benchmark first. And then in terms of integration, we'll have a new blog coming really soon, perhaps like within a week to help provide more information. Thanks. Another question from chat, how long do you expect the benchmark to be given that maybe there's only limited budget? So great question. So I'll start by, I guess, answering this part. So essentially, I think the size of the benchmark is determined by two factors.

So first, of course, the budget. And second is also the question of the investigation scope. It's not always the case that the benchmark is as large as possible. I think as long as it is of a size that can truly do a correct management in a rather stable format, and also successfully support your evaluation target, then that should be fine. For example, just in general for like web browsing benchmark, depending on whether you are testing, say, a general web browsing ability or say if you are just testing for BI-related like website or for let's say like commercial or for financial or medical setting, sort of like website, depending on the problem for scope, I think as long as the size of the data set properly support that scope, it should be fine.

And another question is, how the final assessment will be done, as there are two assessment based on green agent and purple agent? So I guess, like, while the standard will be provided for two phase on the more on our official website, in general, those will be like assessment. There will be like assessed as different tracks and assessed separately. Another question is, how can we extensively test our green agent with LLM API limitations? I see. So in terms of API limitations, so I would encourage to explore, first of all, different, I guess, agent platforms.

And also in general, I think this goes in line with the previous question on the budget for creating the benchmark that I think the purpose is rather just a complete benchmark under the scope rather than just, I would say make it as large or make it as let's say repeat the experiment multiple times as more as possible. So there are some, I guess, in general, public resource for getting more API requests from different LLM APIs provider, but in general, just I guess the principle here is design the experiment more carefully.

And also just look out for more resource. Another question from the chat, do we encourage new benchmark or agentify existing benchmark? I think they'll be measured as-- first of all, this will be measured under different tracks. So we, in general, encourage both of them as both of them will be significantly contributing a rather standardized agent evaluation future. And overall, the assessment will be made-- will be made based on both the effort and also the final outcome for the benchmark. Say for example, there are some-- currently, there are some benchmark that are originally quite simple, but non-standardized.

It can really make them really general, really flexible for different types of agent and also provide, let's say like a logging interface or something helpful agent separation that could potentially lead to very interesting findings or very high utility, then those will be also significantly contribution and counted in the general assessment. And also for new benchmark is the same story, like, it doesn't have to be marking the previous question. It doesn't have to be super large. And as long as it's like novel and satisfied, like most of the criteria that is mentioned in the earlier part of this info session, then yeah, it should be a good candidate for submission.

Here's another question. So why is the evaluator called an agent, won't it be just an evaluator, not an agent? Good question. So actually one of the core part for AgentBeats is that we are treating-- we are trying to unify the abstraction here by also making the evaluator or what we call an assessor, agent entity. And in this way, they actually follow the same set of interface as a general agent based on A2A. So you can actually manage the-- think of it as the following. You can manage the benchmark the same way as you might manage agent themselves.

And that will actually simplify the system development as well as just conceptually a lot. And here, I do want to note that the evaluators or the assessors is not necessarily driven by LLM. So they might be agent, but more towards, I guess, original concept of agents in reinforcement learning that you would react to the environment, which is also like the platform for the request for evaluating other agent. And it's not necessarily driven, or at least for this assessor agents, it's not necessarily driven by LLM. However, I also do want to point out that most of the-- well, significant amount of recent agent evaluation is based on LLM as a judge, which means that LLM component is already somehow within the evaluation procedure.

And under this assumption, we are seeing that in the future, more and more agents will be judged by or evaluated by other agents, and thus we are making the concept or the abstraction to be consistent at this stage. And for the questions, please feel free to leave them in the Discord channel which is the preferred channel. Thank you. Oh, and for the second part of the earlier question, yes, there will be LLM evaluator and there will also be the evaluator-- there will be the evaluator agents as they are also evaluators.

So given the time reason, thanks for the wonderful questions, and we'll be wrapping up here. For more questions, please feel free to leave them in Discord and we'll be replying in a synchronous manner later. Thank you all again for joining today's session, and hope you have fun in building your next agent evaluation. Thank you.

Questions about this episode

Who speaks in "AgentX-AgentBeats Competition Info Session"?

Dawn Song (Main speaker), Xiaoyuan Liu (Co-presenter / technical lead speaker).

What is discussed in "AgentX-AgentBeats Competition Info Session"?

In order: 0:00 Agentic AI MOOC, AgentBeats, Dawn Song, Berkeley Center for Responsible Decentralized Intelligence; 10:00 AgentBeats, Dawn Song, Xiaoyuan Liu, Agentic AI MOOC; 20:00 AgentBeats, Xiaoyuan Liu, tau2-bench; 30:00 AgentBeats, tau-bench; 40:00 AgentBeats, Dawn Song; 50:00 CyberGym, tau-bench, tau2-bench; 1:00:00 AgentBeats, Dawn Song, Xiaoyuan Liu, tau-bench; 1:10:00 AgentBeats.

What is mentioned in this Berkeley RDI Center on Decentralization & AI episode?

8 people, works and ideas are mentioned, including AgentBeats, Dawn Song, Xiaoyuan Liu, Berkeley Center for Responsible Decentralized Intelligence, Agentic AI MOOC, tau-bench. Each one on this page links to its own page with every episode that mentions it.

How long is "AgentX-AgentBeats Competition Info Session"?

The episode runs 1 h 17 min; the transcript is about 9,271 words in 95 timestamped sections.

More from Berkeley RDI Center on Decentralization & AI

About this transcript: the episode's spoken words as captioned on YouTube, split into timestamped sections. Reclip.Me locates every person, work and idea in the text, keeps the time it is first named, and quotes the words around it verbatim. Last updated January 24, 2026.