Transcript

AAAI Presidential Panel Zoom - AI Agents

From AAAI · Episode page · Watch on YouTube

· 1 h 1 min · 10,800 words · 83 timestamped sections

This is the full, timestamped transcript of "AAAI Presidential Panel Zoom - AI Agents", an episode of AAAI published November 18, 2025 (1 h 1 min, about 10,800 words). It mentions 15 people, works and ideas, including Franchesc Rosi, LLMs, Michael Waldridge, Atusa Kazir Zade, AI research.

Who speaks in this episode?

What is discussed, and when

  1. – 10:00: AI research, Triple AI, AI governance, Franchesc Rosi
  2. – 20:00: natural language, Michael Waldridge, AI research
  3. – 30:00: natural language
  4. – 40:00: natural language
  5. – 50:00: natural language, speech act theory
  6. – 1:00:00: speech act theory
  7. – 1:01:28: AI research, Triple AI

Notable quotes from "AAAI Presidential Panel Zoom - AI Agents"

  • AI and AI governance and I'm also the past president of triple AI. And now today I would like to welcome you to the triple AI panel discussion on AI agents.
    about Franchesc Rosi · at
  • More capable and reliable but also more safe. So to contribute to this goal, in March 2025, TripleAI published a report on the future of AI research which summarizes the…
    about Triple AI · at
  • Emerging behavior and so on and also covering the geopolitical scenarios around AI. The report can be found on the triplei website and identified in total 17 topics and for…
    about AI agents report · at
  • Then we have Kartik Talamat Padulla distinguished architect AI at Oracle and Michael Waldridge, triple president-elect and Ashal, professor of foundations of artificial intelligence at the University of Oxford.
    about Michael Waldridge · at
  • Notoriously is notoriously sloppy. And you know the the way that we used to think about solving those problems was ontologies. and we would have very rigorously designed ontologies which formally specify meaning of the.
    about natural language · at
  • Speech act theory has been around for a long time, right? And so, in fact, I see Mike nodding along. A lot of the basis of today's modern theory of multi-agent systems kind of…
    about speech act theory · at

Who and what is mentioned in this episode?

People

  • Franchesc Rosi — Person · Host / main speaker (panel moderator) · 1 mention · first at
    Introduces the AAAI panel on AI agents, frames the discussion via the AAAI 'Future of AI Research' report, and moderates questions to panelists.
  • Michael Waldridge — Person · Panelist / academic historian of the field · 2 mentions · first at
    Gives historical grounding for agents and multi-agent systems (late 80s/early 90s), highlights enduring issues (shared meaning/ontologies), and emphasizes modern security risks (prompt injection) and the role-based structuring of LLM agents.
  • Atusa Kazir Zade — Person · Panelist / subject-matter expert · 1 mention · first at
    Provides a research-grounded taxonomy of AI-agent definitions and connects agent properties (autonomy, efficacy, goal complexity, generality) to governance and risk in multi-agent/LLM settings.
    Futures for AI agents. we will do that with three panelists. the first one is Atuza Kazir Zade, assistant professor at Carnegie Melo University.
  • Kartik Talamat Padulla — Person · Panelist / industry practitioner · 1 mention · first at
    Explains industry perspectives on agent definitions (observation–action/RL framing), practical benefits of LLMs (domain-definition bottleneck), and engineering mitigations (workflow orchestration, closing parts of the world, cost/security considerations).
    Then we have Kartik Talamat Padulla distinguished architect AI at Oracle and Michael Waldridge, triple president-elect and Ashal, professor of foundations of artificial intelligence.

Books & papers

  • AI agents report — Book · Referenced work · 1 mention · first at
    A report summarizing findings on AI research trends, including AI agents.

Companies & orgs

  • Triple AI — Company · Organizer · 4 mentions · first at
    Host of the panel discussion and publisher of the report on the future of AI research.
  • Oracle Corporation — Company · Affiliation · 1 mention · first at
    Affiliation of one of the panelists, providing industry perspective on AI agents.
    Then we have Kartik Talamat Padulla distinguished architect AI at Oracle and Michael Waldridge, triple president-elect and Ashal, professor of foundations of artificial intelligence.

Concepts & topics

  • LLMs — Concept · Key technology · 4 mentions
    Large Language Models are pivotal in the current evolution of AI agents, enabling new capabilities and applications.
  • AI research — Concept · Key discussion area · 5 mentions · first at
    Discusses the trajectory, challenges, and trends in AI research, especially in relation to AI agents.
    Before we get to the discussion on this topic, let me start by saying that as we all know AI capabilities and research are undergoing a very fast and significant transformation…
  • natural language — Concept · Key technology · 7 mentions · first at
    Central to the operation and communication of LLM-based AI agents.
  • AI governance — Concept · Key discussion area · 2 mentions · first at
    Addresses the governance challenges and frameworks needed for AI agents and multi-agent systems.
    Hi everybody. My name is Franchesca Rosi. I work at IBM research and I'm the company's global leader for responsible AI and AI governance and I'm also the past president of triple AI. And now today I would like to welcome you to the.
  • speech act theory — Concept · Referenced framework · 2 mentions · first at
    Highlighted as a foundational theory for understanding communication in multi-agent systems.
  • AI perception versus reality — Concept · Previous panel topic · 1 mention · first at
    One of the topics previously discussed in the panel series, related to AI agents.
    Disciplines, policy makers, funding agents and the media and the general public. We already held in-depth panel discussions on four of these 17 topics and they were AGI, AI…
  • AI and sustainability — Concept · Previous panel topic · 1 mention · first at
    One of the topics previously discussed in the panel series, touching on broader impacts of AI.
    Disciplines, policy makers, funding agents and the media and the general public. We already held in-depth panel discussions on four of these 17 topics and they were AGI, AI…
  • AI reasoning — Concept · Previous panel topic · 1 mention · first at
    One of the topics previously discussed in the panel series, relevant to AI agents.
    Disciplines, policy makers, funding agents and the media and the general public. We already held in-depth panel discussions on four of these 17 topics and they were AGI, AI…

Full transcript with timestamps

Hi everybody. My name is Franchesca Rosi. I work at IBM research and I'm the company's global leader for responsible AI and AI governance and I'm also the past president of triple AI. And now today I would like to welcome you to the triple AI panel discussion on AI agents. Before we get to the discussion on this topic, let me start by saying that as we all know AI capabilities and research are undergoing a very fast and significant transformation along many dimensions including the topics for work, the methodologies, the research community itself and also the working environment. In this rapidly evolving scenario, it's very important to be able to clearly identify the trajectory of AI and of AI research and the challenges that are still ahead of us to make AI more capable and reliable but also more safe. So to contribute to this goal, in March 2025, TripleAI published a report on the future of AI research which

summarizes the finding of a study of several months of significant AI research trends. The study was led by me as the as the then current AI triple AI president but was conducted together with 25 prominent AI researchers and also supported by a community survey with almost 500 respondents. And the topics that were covered in this study were included who is doing AI research more in corporate more interdisciplinary and so on. What new topics is AI research about reasoning factuality AGI and many others. How how AI research is conducted evaluation benchmarking emerging behavior and so on and also covering the geopolitical scenarios around AI. The report can be found on the triplei website and identified in total 17 topics and for each of them you will find the chapter that sketches the history the current trends and the open challenges. The report has been intentionally written in a non-technical

way because the intent was to reach out to many audiences beyond uh AI researchers including experts of other disciplines, policy makers, funding agents and the media and the general public. We already held in-depth panel discussions on four of these 17 topics and they were AGI, AI perception versus reality, AI reasoning and AI and sustainability. And today we will discuss another one of these 17 topics which is AI agents. And AI agents and you will see as you will see during the discussion have come a long way in AI starting from rulebased entities and now evolving in really systems and multi-agent systems powered by a combination of more traditional AI techniques and machine learning and generative AI approaches such as those in large language models. And this evolution has brought a lot of new opportunities, broader application scope, but also raises some challenges.

So during this panel discussion, we will discuss the definition of AI agents their history, the modern notion of agentic AI, the introduction of LLMs in multi- aent systems, issues of governance, risk mitigation and possible futures for AI agents. Uh we will do that with three panelists. Uh the first one is Atuza Kazir Zade, assistant professor at Carnegie Melo University. Then we have Kartik Talamat Padulla distinguished architect AI at Oracle and Michael Waldridge, triple president-elect and Ashal, professor of foundations of artificial intelligence at the University of Oxford.

Let me remind everybody that you can write questions in the chat and uh hopefully we will have time to respond to few of them at the end of the panel discussion. So let me start with uh natural first question about the definitions of AI agents and let me address this question first to Atusa. So there are in the literature many definitions of AI agents at this point. In fact, in a recent paper that you published titled characterizing AI agents for alignment and governance, you mentions seven of them taken all from the computer science literature. So can you help us understand why there are different definitions of AI agents? What are their main commonalities and difference so that we can better identify what are the main properties of an AI agent?

Thank, you., Uh, yeah, as, you, uh, already mentions efforts to understand and characterize a AI agents have a long history. Um if I understand incoring correctly the first time the way the term AI agents was used or term was coined uh was in a 1977 paper by uh Fenbound called the art of artificial intelligence themes and case studies of knowledge engineering. Um so since then obviously with lots of ups and downs and various different periods of intense attention um the topic of AI ages has been uh studied by various different researchers in relation to various different architecture different types of AI systems that were interesting uh at their own time. So in the uh 2024 2025 uh we observed a a surge of uh interest and attention to the notion of AI agents and we figured that unfortunately lots of researchers are treating this uh notion as if it's like a topic that came out since we are

having large language model agents. Um now understanding the characterization and definition of this term is very important because when we want to govern or talk about the governance of AI agents we need to have an operational definition of what these systems are. So we did like a very uh significant kind of like survey and literature review of the of uh the work um on AI agents and we figure that there various different definitions different researchers um according to the leading definitions uh index on specific aspects of agency but there are four main uh kind of dimensions that are core to definitions of AI agency or AI agents and those are uh the autonomy of AI agents uh the efficacy, the causal impacts of the systems on the environment, the complexity of the goals uh that they execute. For for example, some of these systems are capable of long-term planning, some of them are not capable

of that. And also the level of generality at which they are uh they are kind of like executed. Um generality is also very important because if this AI agents have an impact on a very specific domain or sector then we should talk about domain specific governance otherwise we should talk about um uh like cross-sectional governance. So efficacy autonomy complexity generality are very important but again we shouldn't think about these notions as binary notions. We shouldn't think that an AI system is autonomous or not.

There are various different gradations of these notions and various different ways to measure those gradations and I'd be happy to talk about them um as um as you like.

Thank, you., Mike, Kartik,, do, you, want, to add anything? Um, I, I, just, wanted, to, briefly, add, that, I think the report goes into one of those definitions I think Atusa that you mentioned right which comes more from the history of multi- aent systems and how those were defined and so on. But there, is, at least, I, think, at, least definitely in the industry there is also the competing I guess uh or maybe cooperative definition of RL based agents right so the definition of an agent as something that essentially I think it boils down to two things an agent is something that can take actions in the world right and that can accept observations from that world and so it changes its behavior to take into account the observations that come in from that world I don't actually think they're competing I think that both the views work together in some sense but certainly I think today what we're

seeing with a lot of the agentic frameworks and the LLM based paradigm of agents and so on. It relies a lot on this observation action paradigm and I think that's important to remember as well.

Thank, you, Michael. Yeah, well, I, could, say, a, little, bit, about maybe about the history of of agents and multi- aent systems uh and it's really fascinating uh Atusa to see your work and to see how those ideas are interpreted in the modern world. Um so where did where did these ideas come from? So they really crystallized at the end of the 1980s and the beginning of the 1990s. Um and it came from a number of different directions. So I can talk for example about my own story and my own story is that in uh 1987 I spent an internship working for an organization that ran uh the UK branch of the internet and it was in the time in the UK it was called Janet and there was a very small number of universities connected to the UK branch of the internet and I did an internship and I had no experience with computer networks before and at the end of that internship I absolutely understood that um networks

were going to be the future. And at the same time, we were in the middle of the expert systems boom in AI and I became fascinated by AI and I knew that I wanted to do that. And it dawned on me that if the future of computing was going to beworked, then surely the future of AI was going to beworked as well. And so I started to ask myself well, what happens if we have multiple AI systems talking to one another? How are they going to talk to one another? What languages are they going to use? What are the protocols for that going to look like? uh how is one AI program and and we quickly latched on the term agent for that you know how is it going to structure its conversation and then another interesting dimension to that was well what if my AI program for example is not necessarily honest or if your AI program is not necessarily honest how do we deal with that or if these two AI programs that are uh

interacting with one another what happens if they don't share common goals and at the same time there were a number of different ideas that were floating in AI and one really big one and a really important part of that story of agents is the following the historical approach to AI had been a decomposition of intelligence into its component parts so learning planning problem solving natural language understanding and vision and the observation was made by a number of people at the time that you know the field had split into these different areas. How do you assemble those components into an integrated architecture? And that integrated architecture became known as what is an agent. And so the idea was an agent that would have that would integrate all those components of intelligent not just do one isolated thing but would bring them all together. Then there were other ideas as well that were floating around.

There were a AI programs were being used in the end of the 1980s for distributed control scenarios. So where you had two AI programs controlling different elements of a system and again the ideas people began to ask well what is what is actually going on here and it those ideas um those ideas sort of morphed into the idea of agents. So all these ideas came together and by the early 90s they crystallized into what you could call the standard model of agents. And the standard model of agents very much had this idea of independent problem solvers that were typically working on behalf of a user or owner or principal.

a key idea that they would be interacting with other uh agents uh which didn't necessarily share common goals. And the big questions of the time were how do we what is the software architecture for one of these things and how are these agents going to interact with one another? And that by the middle of the 1990s that had launched the multi- aent systems community and that community went on to explore and still does explore issues like well what are those protocols for cooperation coordination, negotiation, what are the languages that agents are going to use and that work is still ongoing but the really fascinating thing about what that we see now is um is the rise of LLMs has just given a complete completely new twist to that story and a new set of capabilities. So for example, when I was busy designing communication languages for agents never entered my head, we'd just be able to do it in natural

language. They just seem a completely fanciful idea. Uh and now we can do that and so many of the issues that we were dealing with we've just leapfrogged over and so this is a really exciting time for this very old idea.

Yeah,, thank, you, Michael., And, in, fact, at this point it really uh after this long history but at this point Aentic AI is really a mainstream area of research and deployment also. In fact many including you know you may have seen open AI CPO declared 2025 as the year of AI agents. Um in our survey in our community survey a majority of respondents found that AI agents are relevant to their research. Many already use AI agents with LLM being the most common technique like 29%. And the survey also showed many application areas that the respondents are aware of such as collaborative problem solving for 68% distributed decision- making think 54% as well as social simulation 41%. So let's start by analyzing what is new now and for now on the positive side. So cartic let's look at the positive implication of introduction of LLMs in multi- aent systems. Are there things and what are these things that we can do now that we

could not do in earlier version of multi- aent systems? Yeah, I think there is I I want to acknowledge Francesca as you're saying there has been a tremendous amount of interest certainly within the industry in adopting these modern avatar of AI agents which is the LLM based LLM backed version of AI agents. Uh and I say interest because interest doesn't always translate into adoption and I think that's where we are now in 2025 and probably in the future as well. I think a lot of this is going to hinge on whether uh you know all of this interest translates into adoption and deployment in real world you know of some of these agentic systems and so forth. But certainly I think from the from a research perspective one of the key problems that LLMs help us overcome uh is is I guess what I would basically call the domain definition curse right so so my background you know my research background some of you know is from AI

planning and one of the big problems we've always had in the field of planning which is as old as you know AI itself has been that you need somebody to come in and give you the expert information you need somebody to define the domain information for a given application domain, right? And I think the, in, fact, if, you you, know, people, on this call who read the report, the report actually touches upon this as well, right? The chapter on AI agents that one of the big problems with widespread adoption of AI back in the '9s and the 2000s and so on was exactly this that there weren't enough real world applications. And I think again that was because of this bottleneck this domain definition curse. I think LLMs today just because of the amount of data that they've been trained on, the size, you know, the number of parameters and what they're able to retain and and generate, they act as very good uh broad

candidate generators, right? Um so regardless of domain, if you go to an LLM and so in fact, I think this is what Mike was referencing as well that you can now program in natural language for example. This is not something that you used to be able to do and you can basically take any application domain you know what have you transportation logistics anything out there and LLMs will give you a reasonably good first pass at things right and so in some sense they are very good candidate generators now is that enough to actually have working real world solutions no obviously and I think in fact some of the past iterations of this panel have talked about the need to verify what is produced by LLMs that's the other important step right once you have a set of candidates, you need to whittle down those set of candidates.

You need to verify which ones are correct. Sometimes you might need to optimize, right? Because some might be more uh uh correct than the others and so forth. But I think that is one of the really positive implications of LLMs that we didn't really have access to in the earlier decades of AI research and development and so on that you always had to have a human or a set of expert humans in the loop defining the domain model of let's say a transportation domain, an airline domain, you know, an insurance domain and so on. I think LLMs do a lot of that work now much faster and for much cheaper. Uh and I think that's one of the big positive implications.

Yeah, thank you. Uh Tuza, do you want to say something more about these positive implications?

Yeah,, I, think, um, some, of, the, positive implications uh could be uh also in the in the field of uh like science. So I've been studying a particular type of AI systems known as autonomous AI scientist systems. And these systems are all by definition um in the LLM uh kind of like space uh multi- aent systems. We don't call them multi-agent systems, but under the hood, they are actually a bunch of like multiple AI agents that are supposed to automate various stages of scientific progress. And they also definitely uh pose um a lot of u possible opportunities for uh for making significant advances in sciences. So I know for example a Japanese scientist um is trying to use this multi-agentic AI systems in order to study various different genomes or like human genomes or gene sequences that have not been studied before. So he thinks that there are almost 70% of human genomes that haven't been like very well studied and

if we like figure out interesting scientific knowledge by studying them in an autonomous way then maybe we can make significant progresses in drug discovery and improving human condition etc. So this could be um like one uh potential uh positive um use case of uh of these multi-agentic AI systems and because again they're working in the natural environment space obviously many more people many more types of scientists also can uh use these types of systems um I'll be happy to talk about the risk of them because I don't want to say these systems are in an unconstrained way uh promising but this could be one really interesting area for positive impact

okay, thank, you, Michael, any, anything, to add to the positive implications of using LLMs in inside a

well, I, think, one, challenge, uh, which remains unchanged and unsolved is something that was um that that was that was subject of a great deal of attention was the issue of sharing meaning natural language interfaces bias an awful lot I mean I say it's it's it's very hard for me to think back to the point where we were designing incredibly crude languages for agents to communicate and then all of a sudden we can just do it in natural language. So that's great, but natural language is notoriously um is notoriously um um sloppy. And you know the the the the way that we used to think about solving those problems was ontologies. Um and we would have very rigorously designed ontologies which formally specify meaning uh of of the term terms that agents are going to use and those ontologies have gone on to be very very successful and one of the very widely um applied symbolic AI technologies. Um but they were designed

for symbolic AI. They weren't designed for the new era of LLMs. But that problem remains. So if my agent is talking to your agent, how do we really know that actually the terms that they're using share a common meaning? Um my my concern is that this is going to turn out to be really quite a big challenge, particularly when we have agents that operate across organizations. Not so much an issue when you have agents that are operating within an organization and we know that they've been designed in the same way. uh um uh by in the same way using the same training data and so on. Uh but I think it really does become an issue across organizations. Can we use ontologies for that? Maybe there is some work on on doing that. I don't know. But but this I think is actually ought to be fairly prominent on our research agenda that how do we know we're really talking about the same things

right?, And, there, are, other, issues, that we now maybe consider in having you know agents that are built by different entities and that or maybe they're on the web available and then an agent goes and finds it and so the open-endedness of these multi- aent systems I think is something new and that raises also the challenge. So now we already passed to the challenges and risk part. So let me start with TUSA. So again we asked in the community survey about the risks of multi- aent agentic AI systems and um the risk that were mentioned by the survey respondents include misalignment between the LLM's general knowledge and the needs of specific systems about 60% they said that the lack of interpretability around again 60% and the security risks around 51% %. So what are more generally these risks of an LLM based AI agent? Some of them as Michael mentioned were maybe already considered as challenges and risk in the previous

versions of agent systems but some of them maybe are amplified or introduced by the use of these LLMs. So and and maybe Atuza you say can the properties of any AI agents these four properties that you mentioned help us also identify the new or amplified risks

right, thank, you, yeah, so, um, yeah, I, as Michael already mentioned uh we already know that um a lot there's been like a lot of challenges um associated with LLM just the LLM themselves or the multimodel variants and we've made a lot of progress in addressing some of those risk and challenges But we have not solved those risk like completely. Like for example, we've made progress in interpretability. We have various different frameworks, various different ways to make sure that we understand what is going under the hood of various different deep neural networks or transformers etc. But there is no one framework that everyone agrees on. These are all work in progresses. Now when we build um autonomous AI scientists uh or like let's say autonomous AI agents um most of these AI agents are supposed to be autonomous we want uh different conflicting things from the systems we want them to be very safe to do exactly

what we want but also they want them to create like new creative knowledge that humans were aren't able to like really create or or um or produce. So um one of the so so then um in short then the the types of systems that we are developing is becoming much more complex and the types of communication issues that very importantly Michael mentioned that existed between different agents uh kind of like uh I think scales up because there's also communication issues between humans and different agents and sometimes we are not even as humans uh we are not able to perfectly clar clarify in a natural language way like what are the requirements that we have from the systems. So there's this very hard like interpretive uh kind of like communication challenge between the way we express things in natural language the way the model um like understands those natural languages, the way the mathematics constraints various

different understandings of the model and I think that is an unresolved problem when we have multi- aent systems. In addition to all the other risk that you mentioned, I think there are three specific types of risks that surface that don't exist maybe when we have just one AI agent system. And there's a report uh that uh was published this year. It's called risk from advanced multi- aent AI systems. And in that report uh with in collaboration with lots of colleagues we explain this. But uh very briefly there are three specific types of risk of harm that could arise from multi-agent AI risk systems. One is the risk of misoperation. So we are not sure if we can we can uh guarantee that the systems co uh like cooperate well with one another. It could be that each system in its isolation is performing its task quite well but then for all of them to come together and perform this holistic task they can they can just

fail. U the second uh problem is uh the issue of conflict uh conflict of interest. So various different systems are supposed to perform task again on themselves like per like correctly. But then there's an issue of um uh optimization multiobjective optimization that has been around since like hundreds of years ago. And that is a problem where for example you want to make sure that the system is uh producing like helpful results but also the results are uh are harmless and it's it it can't you can't have a system that always optimizes perfectly globally for these two systems for these two values. So obviously we can imagine that in a multi-agentic systems we have let's say an AI agent that is supposed to run an experiment and an AI agent that is supposed to produce like perfectly clear s uh papers that can be understood by a variety of researchers and so then there's going to be uh the choice of

simplicity when the when the agents wants to report the results of the experiments and the choice of like complexity when the system wants to run the experiments. And the question is how would the system decide about the right level of tradeoff and the right level of uh value conflict resolution. So miscooperation conflict and the third type of risk is the risk of collusion. Um and so this is uh this is when um again different systems want to optimize for various different targets. Uh but they they they values uh like or they the targets collide with one another.

And again these are specific risks that don't arise when we study as a specific agent in its isolation but when we put the systems together new types of risk uh emerges.

Yeah., Karti, do, you, have, anything, to, add to the challenges additional challenges and risks?

Yeah, I, I'll, just, say, that, I, think, a, lot of the issues that raised are very pertinent issues. Um I don't know just being pragmatic right I don't know that we are ever going to solve all of these issues in some sense because the way that especially adoption of AI agents is proceeding it's less about a you know unified frontier kind of advancing which is usually how you know we kind of look at it from a research perspective but in industry like in in the adoption of these things certain narrow things tend to become more important than other ones for example so to take the example of one thing that are I was saying the rule following, right? That is important right now. Like they, you know, people who are uh putting these agents into real world systems, they absolutely want them to be able to follow rules and they're trying to do everything they can including, you know, prompting guardrails, fine-tuning models that, you

know, these agents are implemented on to to make them very restricted in a specific use case. But I agree. I think there, are, a, lot, of these, issues, because the fundamental uh problem or maybe not a problem but the fundamental uh issue is that we are trying to get these agents to replace human behaviors and to to replicate humanlike behaviors. And of course humans I think are much more complex than any agent that we can create. So I think that's really the the key thing there.

Yeah., Thank, you., And, I, think, that, yes, what Atuza mentioned also this fact that you may try to um reduce or mitigate the risk of a single agent but then when you put them together you get this you know additional risk for and for example another one is related to the issue of data privacy. No, because data is shared among these different agents tools. They're not necessarily all agents. No in the components. And so how you going to handle or do you need to modify the way you handle this data governance or data privacy issue? Because now you have this data shared with with components that you didn't even build. you know somebody else builds and you don't even know the security aspects of the the the privacy aspects and the properties of that component. Um, so is there anything else Mike to add around the new challenges that are posed? Um

I, well, I, think there's, one, key, one, which, I, think, is really really important which is a really new area I think in AI generally and it's it's the issue of security. I mean if we are using LLM agents and we are and the way that we're getting to do the things is by providing them with prompts then we immediately open ourselves up to uh vulnerabilities like prompt injection attacks and at the moment it really seems like um it really seems like we don't have deep fixes for those problems. Um, you know, the there's a there's a a game right now between those that are figuring out how to attack these models and those that are trying to develop guard rails to defend against them. And it's a game of cat and mouse. And I I say we are at the moment not in sight of any really really deep fixes. And that becomes worrying if we're going to have agents that really go out and do things for us. I mean, I'm

going to be concerned if I hand over my credit card number to an agent and that agent then gets hacked. So, this I think has to be uh this has to be on on the agenda. And it's kind of weird. I mean Franchesco, we remember AI from 25 years ago. Security like this was not something I think that we envisaged as an issue at all. We didn't we didn't realize that this was going to be a challenge in the way that it is now. And I can remember, you know, when I first saw the first crude prompt injection attacks on these models and it began to dawn on me, this is going to be a problem. And it still is a problem. And despite all the effort that we've seen go in from the world's richest companies over the last five, six years into trying to defend these models, you don't need a PhD in AI to get around those guard rails. So this I think is going to is is is going to be a a big challenge and it may be one of the make orb

breakak issues for the technology. So Michael, that's related to the input not the prompt that you give to an agent based on an LLM. But then there are also challenges on the output side because the LLM can hallucinate which means that if you give the LLM the task to call another agent or another tool then the LLM could call the wrong agent or they call the agent with the wrong parameters or you know so so uh that is also in the output side where you may have you know calling something that is not what is needed for the task to be accomplished.

So, so that's really really using natural language as all these domain specificity that is given for free in some sense. Um but then you also have these other all these other additional challenges because of the vagueness and of and and the possible hallucin vagueness of natural language but also possible hallucinations and and the prompt injection and robustness security issues and so on. Um so uh but then so then I mean when we go from so we said what LLM can bring in positive or negative terms to an agentic a multi- aent systems uh but then what can agentic system can bring to LLM? So for example, can the fact that you put together multiple components help mitigate these risks with LLMs like hallucination nondeterminism uh reliability, trust. So yes, LLMs have these issues, but maybe putting together other pieces in a multi- aent system can actually help us address those issues.

What do you think, Kartik? Yeah., So, I, I, I, agree., I, think, so, so, I want to go back a little bit to again you know couple of decades maybe 30 years back in AI where if you guys remember cognitive architectures were a big thing at that time right so we had all these things like soore and you know a bunch of other architectures and the idea there essentially was that uh an AI system is not a single model or a single you know component but it is a combination of many different components and in fact even today in the textbooks that all of us learn AI from and so on.

If you look at the prototypical AI uh you know system picture or the AI uh uh agent picture, there are many different components. You know, some are responsible for the vision, some and perception, some are responsible for the actuation, some are responsible for the processing and the planning and the thinking and all kinds of other things. And so the idea was that you would need a cognitive architecture to basically combine all of these different components and to enable intelligent behavior, emergent intelligent behavior.

And I think again the historical uh basis for that is that it comes from trying to implement robotic agents. You know the the SRRI had shaky the robot and strips and all of those things that came out of that. Now I do think that there's something we can learn from that. So a couple of things I want to talk about. One is this whole notion of open versus closed worlds right? So uh in the past uh when we had more let's say deterministic AI systems for lack of a better term uh and also just deterministic programs they were essentially executing on a closed world.

Whereas today I think one of the problems with LLMs is that we are essentially in an entirely open world. You don't even know for example if you will get the same output from an LLM twice even if you give it essentially the same inputs and so forth. And again a lot of it is a black box right we don't really know what's going to come out the other side of that black box. So one of the emergent uh trends that I've seen at least certainly in the industry is LLMs are useful uh to deal with completely open worlds and open scenarios. They are good candidate generators. We discussed this. But once an LLM actually produces a uh an output which you have verified or validated in some way, you should close off that portion of the world. Right? So you have you start with a completely open world and then you close small sections off of that open world because you say that okay this is an acceptable solution it's

above a certain threshold or this is a verifiably correct solution because I verified it with some external thing right and then the important thing the key thing is to not revisit that closed portion of the world anymore right you can still use your agentic you know your LLM based agentic system on the remaining open parts of the world but it's essentially a problem of how do you close off portions of the world and and and I think that also will deal with all these issues of you know uh uncertainty and inconsistency with respect to LLM chains. Uh it will and agent chains. I think it'll also deal with this issue of error propagation. Right? Because right now when we have especially agentic chains when you have multiple agents that are part of a workflow uh and if you know earlier on in the chain you have some error that creeps in that's just going to keep getting propagated down that chain. So I do think that we

can take inspiration from the cognitive architectures of the past where we can kind of wall off almost you know certain specific areas and then we can say in order to close off this section I need an output that exceeds the following threshold right again we're in the world of now probabilities and continuities right it's no longer a discrete world of 01 and you know this is absolutely correct versus absolutely incorrect uh so I do think that and I see a little bit of this happening certainly in the industry, right, now, where, one, of the, ways that people have found to mitigate some of the problems with uh LLM based agents is precisely to embed them in these workflows. Of course, there's also a lot of rediscovery going on in some sense right, of a lot of concepts that were were explored and published 30 years ago or 40 years ago. But again, that's that's part of the challenge. So but uh

this is important point that you make and but I think that a basic point that you also make is that over time you know with LLMs in some sense people thought that you could forget about multi multicomponent architecture you could embed everything inside a single model. Oh no, we don't need to have this and that and that and then deal with the communication and the sharing interaction. Everything is you know you just give me all the information. I train a single model and then the model can do everything. Can I do planning can I do reasoning, can I do this, I can do that. So I don't need anything else just one piece. And now we are rediscovering instead I think with this idea of aic AI you know coming up uh we are rediscovering that we need yes LLM can do a lot but not everything and you need all these other pieces to to get the right reliability the right security the right you know properties that you

want and, and, just, briefly, at, the, risk, of saying something controversial because you know what fun is a panel if somebody doesn't say something controversial the modern AI provider companies, right? We all know who they are. They are kind of almost, you know, it's almost they're finding very inventive ways of saying exactly what we're saying here that, you know, one model is not enough. We need multiple models and so on, right? So for some of them, it's an efficiency play. Oh, we don't want to send every prompt to like, you know, 100 billion parameter model. We need smaller models to do this. For some of them, it's a context play, right? Oh, we can't send the entire context, you know, append everything and send it in one prompt.

So, we want to break it down and so on. But this is, you know, it's the equivalent of kind of going around your head to touch your nose, right? That's not the fundamental issue. The fundamental issue is what you had pointed out, Franchesca, which we've been discussing, which is that one model is not going to be sufficient to solve all of these issues, right? You do need fundamentally different components and you need them to interact with each other. They can all be agents, right? But you need a way to coordinate all of that. So, yeah, I agree.

Okay., Anything, else, to, add, on, this topic? Yeah,, just, uh, that's, fascinating., I, mean I'm I'm learning things uh and giving new ways of thinking. Thank you Kartik. That's really interesting. Um, just to throw in related to that, I mean, I think it's really quite fascinating that we have research which says that LLMs perform well when they play roles and that fits again so well into the the traditional view of multi- aent systems that you have a number of agents playing very specific roles and also I think Francesca the, the the, problem, of, can, you, use, a multi-agent architecture to try to overcome like properties of hallucinations very natural to think about a collection of of agents that are in some kind of uh dialogue where they are challenging one another and demanding uh an exhaustive proof just almost like sort of chain of thought but with kind of multi-agent approaches and

a really interesting thing I think we're at the very beginning of that journey with LLM agents to understand the ways of structuring them but it's really amazing to see these ideas unfold

yeah, so now that we have this picture of these multi- aent systems including LLMs and so on. So what how can we use our knowledge over many decades of AI agents to define the governance the right governance of these systems and does this the right governance depend on the properties of the specific agent profiles like you mentioned earlier.

Thank, you., Yeah., Um, yeah, I, think, I, I want to start again from the place that um from the observation that the governance of LLM's just one model has been like very complex now we are moving to this very complex like a much more complex kind of space what should we do I think one approach to take is like oh we can't resolve this problem so like let's let's just like uh you know pick and uh pick and choose and and go forward I I'm kind of against that approach I think we can actually develop some structured ways of thinking about the governance and I want to go back to the characterization of AI agents across these four dimensions like the dimensions being autonomy, causal impact, uh generality and goal complexity. And so the idea is like okay if you want to develop an AI agent in a particular domain or in relation to a particular deployment area which dimensions we actually really want to

govern and if you ask this question then we would be able to make at least some progress in relation to the governance base. So for example, it could be that we have some AI agents that are like full that are performing fully autonomous actions like think about Whimo like Whimo is an example of an embodied multi- aent AI systems according to according to some people. So for for that particular system we are thinking about the deployment environment. There are specific rules principles developed for transformation for transportation like legal uh domains like different jurisdictions have different standards etc. And we have seen from ITLE E and various different other professional domains that specific standards have been developed around the governance of this particular type of type of a agents. But wayos I mean they they perform complex task but they're also not like reasoning or planning

about like the future future u conversations with like an adult or something or maybe maybe they do or with a ch child or something or maybe they do but if you think about like multi-agentic uh AI companions that are going to be this autonomous systems that can figure out what a specific child or kid or adult wants to do. Seems like now we are in a realm where actually the goal complexity the complexities of tasks that the system performing are also become very important. Then we need to break down the governance questions across this particular dimension of complexity and then think about like the governance in this like more hierarchical way. In the in the case of like uh f like companions AI companions again seems like the systems are able to have conversations across a generality of different kinds of domains. Then we need to figure out okay how we have to consider various different standards

like I read I think was it yesterday or today that a report was published by open AI basically uh giving kind of an statistics about what percentage of the users maybe are having suicidal convers talks with the chatbots that might induce suicidal uh thinking and etc and they're now having conversations with hundreds of psychologists and psych psychiatrists and etc. And so I think this example shows cases where first of all it shows governance is very hard. Second it shows governance requires knowledge of various different disciplines and we have those disciplines. But to govern this very complex AI agents we just need to bridge and bring in those knowledges as much as possible and create new spaces for for governance.

Yeah., Thank, you., Uh, so, Kartik, uh, in, a recent talk that you gave you explained that a lot of work around you know risk mitigation governance and so on and solutions for various issues in a multi- aent systems and and you know much of this today's work on agentic AI system is built on top of work done in the 70 80s and 90s. So is this do you think that this is something that is widely known by people building aic system or is there the risk that researchers and modern researchers companies working and deploying agentic AI kind of reinvent the wheel in the sense of not being aware of the many ways of addressing the issues and studying the issues in the past and the solution that have been uh devised. So with the use of LLM, do we need to start from scratch rethinking about all these or we can reuse and adapt some of some of the existing results?

Yeah., No., So, in, fact, I, I, should, I, should say that I I don't I the answer is no. I don't think people are aware of or even reusing or rebuilding on a lot of the prior work and research that has happened in this space. Right. The the talk that you're referencing and I'm happy to share it with the audience afterwards. Um I was trying to personally make a point right that a lot of what we see today and so the the talk was specifically about today's agentic frameworks right so the think of things like autogen crew AI lang graph you know all of these modern systems that people have adopted quite widely uh in order to design these chains of agents agentic systems and to evaluate them and and so on and so forth to deploy them uh for wide use and so forth and the point that I was trying to make was that a lot of the protocols that these frameworks are inventing are really reinventions of

things that have been around you know for decades at this point. I mean again the point is not that the protocols were around and that somehow you should cite them. It's not an academic argument that I'm making right but rather that we should build on top of it rather than reinvent it from scratch right that's just the more efficient way to do it. So to give you like a a couple of examples on this, right? Uh and I'll go back to something that Mike said about you know the ambi the inherent ambiguity of natural language right we all know this like this is this is why we love you know our languages English famously so right is a very ambiguous language there's you have no clue what somebody is trying to say in English and in fact we have all these different dialects and accents and and so on. Uh speech act theory right this is from Austin and S this is from like the 60s and so on.

Speech act theory has been around for a long time, right? And so, in fact, I see Mike nodding along. A lot of the basis of today's modern theory of multi-agent systems kind of is grounded from that notion of speech act theory and like the philosophy of how agents would even communicate with each other first of all, right? How would they interchange information? How would they communicate with each other and so on? And then, of course, you know, Ray Pero and Cohen and others came up with like uh plan-based philosophies for agent communication and so on. This is all you know stuff from the, ' 60s, and, '7s, that, I'm, talking about. Now why is this stuff important?

It's important because today we see uh you know providers reinventing a lot of these things. So for example, Autogen has this notion of a group chat manager right? What does the group chat manager do? The group chat manager is basically a component that takes all of the verbose output that is generated by the different LLM based agents, right? And it tries to coordinate that group. It tries to coordinate that team essentially, right? to try to push them towards a common goal towards a common resolution and so on. But again the group chat manager is basically the equivalent to the communication facilitator uh in something called KQML.

So there was this whole you know uh interoperability platform called the knowledge management language KQML right you can look it up. Uh so that has existed similarly crew AI they work under this notion of you know agentic crews right groups of agents and so forth. um that again builds on something called the contract net protocol which existed you know 30 years ago and so on right so a lot of this there's been work in the multi- aent uh uh systems communities in in general you know the AI community and I think it's important for us to be cognizant of a lot of that work that's happened if for nothing else then to not reinvent the wheel again right and to look at some of the pitfalls of all of these things and so on what I'm mentioning is a subset but increasingly you know 2025 2026 it may be the year the agent it may not be but what is going to be important is we need to uh have a little bit of

standardization, at least, of, how, these agents are communicating with each other right maybe we don't go to the extent of having entire ontologies and and so on and so forth right but we do need protocols like I think one the key takeaways if I had to say you know two or three key takeaways from the research that I did on this one communication is action especially in today's world of you know a agents and LLM and so on communication is action um design protocols, not just prompts right? We we need ways for these agents to kind of interchange uh information metadata, all kinds of things, right?

And the third orchestration is essential. You have to plan for orchestration and so forth. But anyway let me stop there. I think that's part of the you know lessons from that.

Thanks,, Karthik., Mike,, Mike,, go, ahead. Yeah,, I, can't, tell, you, how, pleased, I, am that we're talking about speech acts and how uh fundamental that was. And I realize a lot of listeners won't won't have won't have been exposed to the idea of speech act. It hasn't been fashionable in AI for a while frankly. But the idea of speech acts is that when we as human beings when we say something we do that typically with some particular goal in mind. So when you know when I when I ask my wife to make me a cup of tea I mean my goal is that she will take on board the intention to make me a cup of tea. And the work that Kartik referred to there, which is beautiful, very insightful work, was trying to understand how we can make precise the various different kinds of speech acts that we all use so that they could use be used by AI systems to plan how to carry out dialogues. And it was very influential early work. And one of

the really remarkable things for me now is LLMs do speech acts. So can we you know if I was I'd love to be a PhD student now uh with all of these problems in front of me can we try to understand the speech acts that LLMs provide and I think your point about about protocols and focusing on those I think I absolutely absolutely uh buy into um that that is absolutely where our um where our activity should be and I just emphasize again this idea of um it really does seem like LLM's perform well when they're playing roles which ties into this idea very very neatly but the there is an entire theory I mean speech act is one component of the theory of pragmatics in linguistics um that, was, as, you, say, was, in invented, by, I mean it goes back as far as victinstein and the idea that victinstein had of language games and then you mentioned Austin and and sir who passed away recently but many others like Grace for

example incredibly influential thinkers about those and they were really trying to understand the nature of human dialogue and can we apply those ideas to understanding you know what large language models do I mean that I say I'd love to be a PhD student addressing those problems now because they'd be such cool problems to uh to look at

yeah, to, that, do, you, want, to, add, anything yes, I, I, love, this, conversation, because um like when I started looking at the problem of LLM alignment in 2020 I I just thought Look, this is a sort of a conversation between an artificial creation and humans. And so why researchers who are developing the systems don't rely on speech act theory? It's it's so shocking and I just want to again as Cartik said highlight that uh for some reasons maybe various different reasons uh lots of more recent AI researchers um and of course not all of them have become so ahistorical. So it's as if nothing existed as if no uh discipline has reflected on ideas of good communication, meaningful communication actually whether LLM should be able to uh produce speech acts, what types of speech acts are there, maybe LLM should not uh output performative speech acts for example if they are uh having conversation with

children, things like that. So like really trying to think about these systems uh in a much more theoretical philosophical way rather than just trying to measure something about them that is not rooted in any kinds of like conceptual uh work wouldn't maybe makes us or bring us as significant advances that we wanted. So yeah this idea of like pragmat pragmatic pragmatism that is coming from philosophy roleplaying uh speech acts communication theory I I I just want to give a shout out that those ideas were very important for alignment of LLMs very important for building successful multi- aent systems as well.

Yes. And and uh in some sense I think that uh the fact that these LLMs are trained over uh uh text that covers every discipline no covers everything. Um in some sense maybe I think makes people think that if there is something to learn from other disciplines is there and the LM will have learned it in the training. No, so we so in some sense that reducive it makes people perceive as if it's less needed to go and look at other disciplines and what what they can bring to improving AI into improving this you know machinery that we embed into machines because in some sense the embedding of cultures knowledge over all the disciplines is is already there you know in in in the training data. So um yeah, it's also true that the amount the speed and the amount of papers and results and announcements is so overwhelming and so fast that people is they're so overwhelmed by is what's happening every day. Then maybe they

have less time and going and look at all the things that have been done in the past that can be useful if adapted you know uh to the current uh scenario. Um so the overwhelming amount of information I think it's also an impediment to this need for exploration and need for looking back at what was done earlier. Um so let's uh finish with some final thoughts from all of you about the future of of agentic AI systems. So now we looked at the opportunities, we looked at the risks the challenges, we looked at the past the present. So but h are you optimistic pessimistic? What do you think the future of AI agents uh is at this point?

Yeah,, Mike,, go, ahead. So, I, think, firstly, there, is, a, lot, of snake oil being sold out there um with with really some really trivial and uninteresting work being being um being sold as as agents. But I think there are some some deep applications like the the one that I really loved was the I think it was deep mind work on the AI co-scientist where you know an assistant in a scientific problem was organized as a collection of agents each playing a role exactly as we talked about earlier. So I think there is uh there there is real work there. I think what we're going to see is a spectrum of agents from agents that are kind of agents like customer service agents right through to those applications like the deep mind one and I think the the really interesting for me sort of scientific challenges and the excitement for me is at that is at the top end of that spectrum but there is going to be a huge

amount of benefit to be obtained by building agents across that spread of that spectrum. Um, and I think the last thing I'd say is, as I say, I right back as far as the 1980s, I got the idea of agents. It seemed to me to be a natural development of idea in ideas in AI. It's just seems to be just as relevant today.

Thank, you,, Atusa. Yeah, I think um it's very important to kind of try to play around with uh some of this uh like the the most advanced versions of this multi-agentic AI systems in particular domains and figure out their limitations and their opportunities. So for me one of the very interesting areas is uh again looking at the instances of uh autonomous AI scientist systems which is actually a type of systems that people have started looking at since like last year in the context of like LLM agents and they're trying to figure out where and how this these systems fail and whether it there is always a necessity for having like expert humans or careful humans like in the loop or not. And I think just playing around with the particular types of systems and trying to look at the risk and and benefits would really help us to make advances in the in the field of uh multi-agna system the next couple

years. Thank, you, to, the, Kartik, final, words. Yes,, I'll, I'll, be, brief., I, know, we're almost out of time. Um couple of things that I think we need to fix in order to get AI agents actually again like I said to go from interest and interest towards adoption, right? One is this notion of like role-based access control. We really we we mentioned this a little bit but if you want agents to engage in fundamentally complex behavior that humans have shown you need to also give them that filter that they will not expose certain kinds of data or certain kinds of information depending on the audience depending on the control. So that's one. The second uh is in terms of the cost, right? So the cost of running these agents today is prohibitive right? And I think that can be tempered down quite a bit, right? Also because a lot of things that we are using agents for today, it's just wasteful. You don't

need to call an agent for a deterministic flow once it's established. You know, you can just execute that deterministic flow on a CPU. You don't need to call an agent that's running in a data center in, you know, Nevada somewhere, right? So that's the second thing I think cost is and when I say cost I mean you know electricity water environmental considerations all of those things rolled in right that's the other thing and I'll end with this I do think that AI agents and agentic workflows they have a lot of potential to greatly increase the productivity of humanity at scale right we just need to direct it toward the problems that matter right as Mike said we need to kind of be able to separate the snake oil which which there is a lot of right now in the industry from the really important problems today the problems that we are gated by unavailability of expertise or you know enough humans to do something and so on

if we can apply agents towards that I think we can see a great productivity boost whether we'll be able to do it is the big question

thank, you, Kartik, so, uh, I'm, sorry, that, we didn't have time to go through some of the questions from the audience but I checked some of the comments and I think we covered some of the issues and some of the topics that they were mentioning. So let me finish by thanking all the panelists for very insightful discussion where I learned a lot. I hope you all also have learned um and given also some good food for thoughts on AI agents. Thank you also to all of you uh that are here joining us. Uh the panel has been recorded and the recording will be available soon on the triple AI website.

the same website where you find the future of AI research report and you can also find the recording of the previous panels. Um, and it will also be available soon on the triple AI YouTube channel. So, thank you all and see you soon for another interesting discussion on AI. Bye-bye.

Questions about this episode

Who speaks in "AAAI Presidential Panel Zoom - AI Agents"?

Franchesc Rosi (Host), Michael Waldridge (Panelist / academic historian of the field), Atusa Kazir Zade (Panelist / subject-matter expert), Kartik Talamat Padulla (Panelist / industry practitioner).

What is discussed in "AAAI Presidential Panel Zoom - AI Agents"?

In order: 0:00 AI research, Triple AI, AI governance, Franchesc Rosi; 10:00 natural language, Michael Waldridge, AI research; 20:00 natural language; 30:00 natural language; 40:00 natural language, speech act theory; 50:00 speech act theory; 1:00:00 AI research, Triple AI.

What is mentioned in this AAAI episode?

15 people, works and ideas are mentioned, including Franchesc Rosi, LLMs, Michael Waldridge, Atusa Kazir Zade, AI research, Triple AI. Each one on this page links to its own page with every episode that mentions it.

How long is "AAAI Presidential Panel Zoom - AI Agents"?

The episode runs 1 h 1 min; the transcript is about 10,800 words in 83 timestamped sections.

More from AAAI

About this transcript: the episode's spoken words as captioned on YouTube, split into timestamped sections. Reclip.Me locates every person, work and idea in the text, keeps the time it is first named, and quotes the words around it verbatim. Last updated January 24, 2026.