AAAI Presidential Panel on Factuality And Trustworthiness

Who speaks in this episode?

• Franchesca Karosi (Host)

What is discussed, and when

1. 0:00 – 10:00: Triple AI, IBM, Franchesca Karosi, AGI
2. 20:00 – 30:00: Joe Biden, 2025, New Hampshire primary, NIST
3. 30:00 – 40:00: AI attribution, Watson
4. 50:00 – 1:00:00: Granite Guardian
5. 1:00:00 – 1:03:55: Triple AI

Notable quotes from "AAAI Presidential Panel on Factuality And Trustworthiness"

• Hi everybody, my name is Franchesca Karosi. I work at IBM research. I'm the company global leader for responsible AI and AI governance and I'm also the past president of triple AI. And I would like.

  about Franchesca Karosi · at 0:00

• Of us to make AI more capable. but also reliable and safe. To contribute to this goal, in March 2025, TripleAI published a report on the future of AI research which summarizes…

  about Triple AI · at 0:48

• To go deeper into each one of these topics, TripleAI already held in-depth panel discussions on five of the 17 topics. AGI, AI perception versus reality, AI reasoning, AI and.

  about AGI · at 2:01

• Orurin. Henry KZ, professor of computer science at the University of Virginia and Charlottesville. Hi Andry. And Kush Vashny, IBM fellow and co-director of the IBM science for social good program.

  about IBM · at 3:35

• Document or some text or some analysis, they're much more likely to be persuasive. We've struggled with well-c calibrated confidence, right? So basically the system is always…

  about ChatGPT · at 19:24

• Of the good actors that want to do the right thing and be transparent about it and so forth. So we've actually created this AI attribution sort of mark that it's not just yes…

  about AI attribution · at 35:00

• Run which is a bit crazy but you can do some sort of model risk evaluation there's even like one of the IBM products called Watson extra governance I mean we'll do this for you there's other open- source things there's Helm from Stanford.

  about Watson · at 37:55

• So I think there is a need for things outside of the model itself. So yeah I mean one thing we've been developing is this model called the granite guardian which is able to do…

  about Granite Guardian · at 55:00

Who and what is mentioned in this episode?

People

• Franchesca Karosi — Person · Main speaker (host/moderator) · 1 mention · first at 0:00
  She opens the panel, frames the topic (AI factuality and trustworthiness), and cites her leadership roles at IBM Research and AAAI/TripleAI, including leading the March 2025 future-of-AI-research report.
• Joe Biden — Person · Referenced public figure (example in deepfake discussion) · 2 mentions · first at 23:26
  Mentioned as the subject of a widely discussed AI voice deepfake robocall used to illustrate how persuasive and hard-to-detect synthetic media can be, impacting trust in information ecosystems.
  Prominent case, and I'll just mention one, but there are many others, was the fake robocall of Jo of Joe Biden's voice in the New Hampshire primary, right? Where people could…

Companies & orgs

• Triple AI — Company · Business focus · 4 mentions · first at 0:08
  Organization hosting the panel discussion and publishing reports on AI research trends.
• IBM — Company · Affiliation · 2 mentions · first at 0:03
  Employer of the panel host and a participant in AI research discussions.
• University of Washington — Company · Affiliation · 1 mention · first at 3:29
  Academic institution associated with one of the panelists.
  Research. We will discuss both these topics and much more with the following panelists. Orini, founder of true media.org and professor emeritus at the University of Washington.…
• NIST — Company · Referenced organization · 1 mention · first at 26:55
  Mentioned in the context of AI detection competitions and programs.
  Been work in our lab also. So I mean Pinu Chen and others have been developing these detectors and I mean NIST has been running I mean all sorts of competitions and programs…

Products & tech

• ChatGPT — Product · Referenced product · 3 mentions
  Example of an AI system discussed in the context of factuality and trustworthiness.
• Watson — Product · Referenced product · 1 mention · first at 38:03
  IBM's AI product mentioned in the context of governance and factuality evaluation.
• Granite Guardian — Product · Referenced product · 1 mention · first at 55:08
  IBM's model mentioned for hallucination detection and other AI trustworthiness evaluations.

Concepts & topics

• AGI — Concept · Core topic · 1 mention · first at 2:10
  One of the key topics discussed in the panel, related to the future of AI.
• AI perception versus reality — Concept · Discussion topic · 1 mention · first at 2:10
  One of the topics covered in the panel discussions organized by TripleAI.
  To go deeper into each one of these topics, TripleAI already held in-depth panel discussions on five of the 17 topics. AGI, AI perception versus reality, AI reasoning, AI and.
• AI and sustainability — Concept · Discussion topic · 1 mention · first at 2:14
  One of the topics covered in the panel discussions organized by TripleAI.
  To go deeper into each one of these topics, TripleAI already held in-depth panel discussions on five of the 17 topics. AGI, AI perception versus reality, AI reasoning, AI and.
• AI reasoning — Concept · Discussion topic · 1 mention · first at 2:14
  One of the topics covered in the panel discussions organized by TripleAI.
  To go deeper into each one of these topics, TripleAI already held in-depth panel discussions on five of the 17 topics. AGI, AI perception versus reality, AI reasoning, AI and.
• AI attribution — Concept · Discussion topic · 1 mention · first at 35:05
  Mentioned as a method to provide transparency about AI-generated content.

Places & events

• 2025 — Event · Referenced time · 2 mentions · first at 0:54
  Mentioned as the year when TripleAI published a significant report on AI research.
  Research and the challenges still ahead of us to make AI more capable. but also reliable and safe. To contribute to this goal, in March 2025, TripleAI published a report on the…
• New Hampshire primary — Event · Referenced event · 1 mention · first at 23:30
  Mentioned as an example of AI-generated disinformation impact.
  Because this isn't just people going to an AI system and getting perhaps incorrect information. One very prominent case, and I'll just mention one, but there are many others…

Full transcript with timestamps

0:00 Hi everybody, my name is Franchesca Karosi. I work at IBM research. I'm the company global leader for responsible AI and AI governance and I'm also the past president of triple AI. And I would like to welcome you all to this triple AI panel discussion on AI factuality and trustworthiness. But before we get to the discussion, let me start by saying that as we all know AI capabilities and research are really undergoing a fast and very significant transformation along many dimensions including its topics, methods, research community, the working environment. And in this rapidly evolving scenario, it is important to be able to clearly identify the trajectory of AI capabilities and AI research and the challenges still ahead of us to make AI more capable. but also reliable and safe. To contribute to this goal, in March 2025, TripleAI published a report on the future of AI research which summarizes the finding of a study

1:03 of several months on significant AI research trends. The study was led by me as the then current triple AI president and conducted together with 25 prominent AI researchers and supported also by a community survey with almost 500 respondents. Topics covered in the report include who is doing AI research? What new topics in AI is AI research about and how AI research is conducted and also mentions the geopolitical scenarios around AI. The report which can be found on the triplei website identified in total 17 topics and for each of them the report includes a chapter that sketches its history, current trends and open challenges. The report has been intentionally written in a non-technical way to reach out to many audiences including experts of other disciplines, policy makers, funding agencies, the media, and the general public.

2:01 To go deeper into each one of these topics, TripleAI already held in-depth panel discussions on five of the 17 topics. AGI, AI perception versus reality, AI reasoning, AI and sustainability, and AI agents. And today we will discuss another one of these 17 topics, AI factuality and trustworthiness. So, an AI system is factual if it refrains from outputting false statements, the so-called hallucinations. And of course, it's difficult to trust an AI system if it can be non-factual. However, trustworthiness extends factuality to include other criteria such as human understandability, robustness, and the alignment with human values. According to many company executives, lack of trustworthiness is one of the main obstacles to adopting AI at scale in their organization, especially in critical application. So improving the factuality and trustworthiness of AI system is arguably the largest area of AI research today.

3:10 Indeed, over 75% of the respondents to our survey strongly agreed that factuality and trustworthiness were relevant or very relevant to their own research. We will discuss both these topics and much more with the following panelists. Orini, founder of true media.org and professor emeritus at the University of Washington. Henry KZ. Hi Orurin. Henry KZ, professor of computer science at the University of Virginia and Charlottesville. Hi Andry. And Kush Vashny, IBM fellow and co-director of the IBM science for social good program.

3:48 Hi Kush. Before we start with the questions, let me remind everybody that you can write your own questions in the chat window of of of the of this call and we will try to incorporate in the discussion some of the questions if there is time. So let's start from defining factuality and trustworthiness. So Orin I will start from you. How would you define factuality in AI and what does it have to do with hallucinations and deep fakes? I I think that uh the definition you shared is uh accurate and I think that the the the key point is that um we come to expect certain principles when we interact with computer systems, when we interact with people uh that they can be trustworthy, they can be reliable. When you ask uh what is the capital of France and you hear Paris, that's a fact. When you see a citation to a resource, okay, the uh population of Poland is this number based on this survey, that

5:03 citation is accurate. These are just kind of basic things that we as non-experts in AI tend to assume. And the problem is that and this has been documented. their famous cases, lawyers submitting briefs, right, that contain these uh hallucinations or erroneous citations. Those sorts of things are are notorious. And the issue is that we have systems that don't have them. So, sorry, I'm going a bit beyond the question, right? But you define factuality. Basically, uh factuality is you get what you expect in objective terms. And the issue that we're here to discuss is at least in recent times uh RAI systems failed to deliver that.

5:49 Thank you Aron Kush. Andrew do you want to add anything to to this topic? Go ahead Andre I was just going to say that um uh in in that first question about hallucination and deep fakes. We can think of a deep fakes as the user intends uh to deceive and hallucinations problem is the system unwittingly deceives itself.

6:17 Yeah. Yes. Yeah. I was going to add and I think we'll get into it a lot um later is uh I mean there can be different truths, different facts, different versions of uh kind of what people believe. Um, so I mean objective is also kind of a a strange word sometimes, but um, uh, yeah, I think we'll we'll get into it as we go along.

6:40 Okay. Yes. Um, so then from factuality, let's go to trustworthiness. So Kush, what does it mean to trust an AI system? What properties should this system have to be trusted beyond even factuality? [snorts]

6:57 Yeah. So I mean Orin kind of started saying this also which is I mean humans can be trustworthy um AI systems uh can be trustworthy in in different fashions but at the end of the day I think they're kind of more or less the same and the way I like to explain it is like think about if you're going to hire a carpenter to work on your house um like what makes that other person trustworthy and um there are kind of these different attributes. One is that u they're competent at what they're doing. I mean, they're doing what they're they say they're going to do. Um, but they're also reliable. They kind of are able to keep that competence in different settings, different conditions, and so forth. Um, also that you can communicate back and forth with the uh with the other person. Um, can uh they understand you, you understand them, and so forth.

7:44 And then they're kind of selfless. Um, so they're working for your goals rather than for for their own goals. And all of these, I mean, completely translate over to AI systems. And um when we're talking about traditional machine learning sort of things um the general competence becomes things like accuracy when it's um the reliability it becomes kind of robustness both um adversarial and um kind of general robustness but also fairness so that things work for for different people different um uh sort of groups individuals and so forth and then kind of explainability and um kind of yeah I mean these sort of things all kind of become become part of Good

8:24 Henry Orin anything about trustworthiness any other property well I I think one one aspect of trustworthiness is that it's essentially it's in the head of the user right and um uh well actually it's a combination a user can trust a system so that's part of trustworthiness at the same time the system might be not trustworthy so not worthy of trust. So the system both has to be worthy of trust and inspire uh trust and and we you can come up with examples that go each way where a system is very uh trustworthy but people still don't trust it. And that's an issue with for example uh you know even even early uh work on medical expert systems for specific domains like bacterial infection where the system was very trustworthy but it did not inspire trust

9:26 and and I think it can happen also the opposite that the system is perceived as being trustworthy but maybe is less so you know and people

9:37 exactly over relly, over delegate, over trust. Right. Right. And and when you think of, you know, why do we trust a person is they've not only let's say done good work for us in the past, but they can explain, you know, why they're making the decisions they're making. And so that's why then this notion of trustworthiness includes um explanation. Of course, you can have an untrustworthy system that comes up with bogus explanations. So, it's kind of a a recursive process.

10:15 Yeah. So, uh so Henry, but then uh Kush also mentioned robustness. So, in your view, what what is AI robustness in and how does it relate to trustworthiness? Yeah. So I I look at Rosebotus as as sort of the opposite of fragility. And we think of an AI system as fragility when it fails catastrophically once it's operating outside of its intended uh parameters. So a system is robust uh both if um uh let's say part of it is breath not being quite so narrowly focused but a large part of it also be simply being able to recognize when it's out of its depth and it can come back.

10:59 So you might have a robust um might have a system that can give you advice about cancer and um the a robust version might say if you're asking questions about things that don't involve cancer then it it tells you you know you're out of my area of expertise where a non-ro system just starts to uh you know diagnose you with every kind of cancer in the world because you So it it nothing relates your symptoms to any of its uh knowledge.

11:37 Orin or Kush, do you want to say more about tr uh the relationship between robustness and trustworthiness?

11:44 I I I just want to highlight again I think this is obvious to the audience and I see some questions coming up in the chat that are very tantalizing but I won't address them until you you tell us to. But I I just want to highlight the the fact that uh the profile of these errors is so different, right? So uh and this is very much tied to this point about resilience, right? And this person who's done great work for you before, you tend to trust them again. An AI system that's done great work for you before is not any more trustworthy because of what's called the the jagged edge, right? the very next question or even a paraphrase of the same question can yield a a radically uh less good answer. So it's really important we tend to anthropomorphize technology all of us even AI experts and it's really important not to fall into that trap and and not do that when it comes to these systems. So the heristic I use is never

12:46 trust an AI system and we might get into some debate about that. uh maybe down the road which will be fun but I uh firmly do not trust AI systems.

12:59 Yeah. And um yeah just one since we're in this definitional part of the conversation. So another term that we hear a lot about is uh safety and um different people kind of define it differently. So I just wanted to kind of give my definition. Um so it is actually I mean like these sort of things right. So it is um uh this just general reliability for unexpected sort of harms that might come up and doing well um against uh like the expected ones as well. So kind of just that in my mind is what safety is no matter what field we're talking about but uh especially and including AI. So just wanted to to throw that in there as well.

13:42 Okay. Uh so now let's see how we got here. You know if these these issues are new issues or or not. So uh Henry when have these issues been raised first in the history of AI

13:58 right? Well, I think they came Yes, sir. you know, very early in the days of expert systems and they came around the idea of understandability that a um a user would not be able to trust um a a system that they could not understand. And and so uh even the very early expert systems often had um an explanation component that would try to give a summary of its reasoning. You can argue about how how useful those um those uh summaries were um in terms of of giving correct answers as opposed to just inspiring a sense of trust. Um I think you know we saw there the the shift from uh purely rulebased systems that could have more cat catastrophic failure modes uh to basian systems where um the the result would be again now more robust in the sense that you're going to get um a probability distribution over answers. People may or may not like that, but at least I think you could come back and say, "Okay, now

15:13 I can quantify um uh each of the, you know, there's five different possible conclusions and leave it up to the user uh to pick between them."

15:24 Kush. Yeah. No, I think that did uh I mean that was a great answer. Okay. But then Kush uh so Okay. So we looked at rulebased systems deterministic some of them probabilistic and you know introducing some nondeterministic but then are these issues uh some part of these issues are there specific to generative AI so not even m more traditional machine learning but generative AI or where they also present in other kinds of AI system because after all even non-generative machine learning system can make mistakes no they always have an error rate and uh so what uh issues related to factuality or trust are new or amplified by generative AI?

16:17 Yeah, I think that's a great question and let me I mean answer with the same sort of um dimensions I mentioned earlier. So when I was saying kind of the basic competence or basic performance with traditional machine learning as you said they can have an error rate right um uh so there's some ground truth that you're comparing to but with the generative AI I think it starts becoming more about uh what people call helpfulness um and then like kind of honesty or like the factuality as we've been talking about so this helpfulness is like hard to characterize there's no like precise ground truth to it but uh maybe there's um like u vibes or some other ways that you can kind of get a sense of is this um uh generative ailm sort of thing being uh being helpful or not. And then on the um uh reliability side of things um uh yeah I mean you don't want there to be um kind of harmfulness. So generating like all

17:14 sorts of bad things whether it's discussions of uh violence or hate speech or um uh kind of like jailbreaking sort of things or I mean all sorts of things that can be harmful. So like how do you um uh have these things be harmless is then the the main point there. And then on the explanation side of it, I think um uh like before it was like truly like what was the reason that this model made its prediction. But um with the generative AI, I think the more important questions sometimes becomes um can you attribute the output?

17:49 So what output tokens like where did they come from? Was it something in the prompt? Was it something in the original training data in the fine-tuning data? these sort of things. And um on the selflessness side of it, I mean uh I think it's much more about um expressing values um and we'll get to to that a little bit later, but um yeah, like can we actually align these things to to human values or yeah societal values and stuff. So becomes quite uh much more I mean psychological in some sense rather than like how just a comparison of uh of ground truth to to outputs and so forth.

18:25 Okay. I wanted And yes, I wanted to ask you because with generative Yeah. I think this tendency to anthropomorphize is amplified, right? Because they they write in our own language, right?

18:40 That's right. I I think there's a lot to say here and and your question is very much on target. First of all, right, when we're doing something discriminative, we typically have a hypothesis and we say, okay, does this person have cancer or not? And we typically have well-c calibrated confidence. So if the system says the person has cancer based on past performance etc as long as who's within distribution we can say okay this answer is likely to be true with 85% confidence etc etc in the generative case right this is very very different and I think one way to be very concise about that is to say that the systems are a lot more persuasive right they're generating some uh document or some text or some analysis, they're much more likely to be persuasive. We've struggled with well-c calibrated confidence, right? So basically the system is always highly uh confident and you see this when you

19:38 interact with chat GBT and then you point out that it's completely incorrect like it failed to count the number of Rs in the word strawberry and it said, "Oh, you know, I'm sorry. I'm sorry, but it it you're right, but it's, you know, it's remarkably confident." And then the last point has to do with the explanation, right? Part of the persuasiveness is they can give you a very persuasive explanation that can be completely wrong as well. So the net effect is that even though machine learning discriminative systems in the past were not 100% reliable, we are now in a completely different realm where basically right these systems are kind of demagogues. They will espouse something with great confidence with excellent explanations be very persuasive and dead wrong.

20:26 Yeah. But also don't you think that the fact that they are general purpose in a sense? So the they are perceived as system that know everything about you know every about every topic. While with more traditional machine learning, you knew that it was built to answer certain questions about some specific topic and so you knew that it was not to be trusted if you asked something completely different. So here you don't have that perception anymore. You actually feel that you can ask anything about any topic and the system as you say responds with great confidence in any case. Right? So that also adds to the perception of being trustworthy when maybe you know you have to be careful.

21:14 Enry, do you want to add anything to the new issues or amplified issues with generative AI?

21:21 No, I I think I think people have blessly touched on all the major points there.

21:27 Okay. So now uh let's try to understand whether we can at least assess this factuality or trustworthiness even if we can we cannot mitigate it but let's try to assess it or um so let's start with orin and with deep fakes. So deep fakes related to images, videos are at this point very very realistic. But is there a reliable way to recognize whether an an image or a video is fake? That is is generated or modified by AI.

22:06 Right? So [clears throat] images, video and audio here are a kind of a different species of this uh unreliability uh different than what we sometimes used to thinking about in uh AI, NLP and so on as you said right they're very persuasive I studied this very closely in 2024 uh in the context of all these elections all over the globe and even back then uh there were a number of studies that we did that other people did and published. There were uh quizzes uh published online for example by the New York Times that showed that people are not better at detecting whether say a face or or an image is fake they're not better than flipping a coin. So basically uh even though people think they can tell they they cannot since then right it's gotten worse in the sense this is a fast-moving field a number of tools from Google openai elsewhere have been uh released and and I want to mention that uh this

23:11 has very very uh important consequences because this isn't just people going to an AI system and getting uh perhaps uh incorrect information. One very prominent case, and I'll just mention one, but there are many others, was the uh fake um uh robocall of Jo uh of Joe Biden's voice in the New Hampshire primary, right? Where uh people could not tell it sounded I couldn't tell. It sounded just like him if you haven't heard it. And it was supposed to be a phone call, right? So, it's imperfect uh acoustics and you just couldn't tell that it wasn't Joe Biden calling you. I mean not personally calling you but you know a message from uh President Joe Biden. So it just shows you how this issue right proliferates and goes way beyond uh kind of access to sophisticated AI systems. It's basically this sort of uh deep fakes being propagated via social media and showing up literally on your phone or on your screen uh in a way that

24:17 uh people cannot defeat. So the answer would be no. You're saying there is no reliable way to recognize.

24:26 Yes. I'm sorry I was getting carried away. Summarize summarize what you said. The the answer is a resounding no. But uh uh like few years ago you were you had a system that could that succeeded in recognizing at least part of a picture as generated or modified by AI. Is that are those techniques not you know not working anymore because of the new more realistic capabilities of AI or what's the problem?

25:03 Sure. So again let me quickly tell the the story. So there are a lot of claims made in this field

25:10 and what I was referring to when I said is people cannot tell. So again the studies that I was referring to were people looking at and say I can tell this is fake or not and they just cannot. Now um we built a very simple mixture of experts model that ran some of the best commercial systems, some of the best open- source models tuned to this and then collected them uh together to to make an assessment. We found that in certain cases in 2024, we were able to tell with high confidence, not 100% sure by any means, but certainly much better than random by uh collecting all these tools, all these experts. Uh we've not studied this uh in 2025 where there have been a lot of advances. So the trajectory of this is very negative for being able to tell even using automated tools. So I can't give you a quantitative answer that's up to date.

26:12 But the bottom line is that uh people can't tell and the tools are getting worse and worse and so we need different kinds of solutions and people have looked to provenence to digital watermarking. We can get into a lot of details there, but the bottom line is we're in bad shape when it comes to detecting deep fakes.

26:32 Yeah. Okay. Thank you. Anybody wants to add some anything? Yeah. I mean, just uh reiterating what Orin said, I think um this is very much a cat and mouse game and we're losing um over time. So uh yeah, I mean there's been work in our lab also. So I mean Pinu Chen and others have been developing these detectors and um I mean NIST has been running I mean all sorts of competitions and programs and and so forth but yeah I mean you're successful at one point in time but then uh you wait a couple months and uh there's a big question is that going to stick around or not?

27:12 Okay. So uh so now so let's go to um LLMs rather than uh image or video sound generators. So Henry to transworthiness applied to uh the content generated by LLM includes also the ability to recognize if a piece of text is generated by AI. So in this case the trust is on the AI system but also on its users and this is what happens for example when authors or reviewers as you know very well of AI research papers that generate papers and generate reviews using AI or even students that write their essays uh with AI. So is it possible in some way to recognize if a text has been generated by an LLM? Are there tools that we can use to that to do that?

28:09 There are tools, but unfortunately they're becoming less and less effective. So, uh, there certainly are are a wide variety of open- source and and, uh, commercial tools that you put in an essay and on a lengthy piece of text, it'll try various um, uh, heristic even use other LLMs to try to guess whether it's uh, likely to be LLM generated or not. Of course, the the cases you read about in the news are where a human wrote it and it said it was LLM. Um uh I assume a lot of the cheating you know when students cheat it's it's so blatant um that uh you know it's it probably the tool is not nearly as useful as simply uh recognizing you know the student who is failing out of the class in other ways couldn't possibly have have written this. I I think the the problem though is becoming worse. Uh partly because as LLMs become more fluent writers, we know there's a real crisis in education and students

29:19 are becoming less fluent writers. And uh they're going through, you know, I remember in high school, you know, sweating over uh essays about Shakespeare, you know, writing a 10-page essay about Shakespeare and and revising your eyes. And if you've never done that in your your your life, uh uh you're you're just going to be a terrible writer probably and not be able to express yourself clearly. Um and so then then what are you going to be comparing the AI against? Um and and I guess the maybe perhaps the best we can hope for in the future is um uh you know AI assisted writing so that you can make sure at least the students have have some uh some input. Um but certainly I know at at the university a a lot of professors are uh moving to things like um uh you know ex written exams essay based written exams in in the classroom where there's no electronics and they have to hand write and um of course

30:34 that's hard to do if you have hundreds of students. Uh students hate it. It's not easy to grade all those essays, but um you know and and I think you know universities probably don't like necessarily discovering that they just admitted a bunch of students who are illiterate.

30:53 Yeah. Yeah. And of course the the impact goes beyond not being able to be a good writer. No, but not being able to articulate well your thoughts in your head and then to to to discuss with others to compare to to compromise. So it's really goes beyond the ability to write a good say uh if you bypass it by asking you know an LLM. Yeah. And and I guess the one point though that people also worry about alms make it easier to generate disinformation and I'm less worried about that because uh it seems pretty easy to get these sweat shops full of of people to write disinformation. Um that you know organized crime gangs and hostile governments are doing that already. So I guess LLMs could make that worse. But it almost feel like that's a different problem.

31:49 Well, right. But I I do want to highlight um that um of course you're right, Hen. You're right. People have been uh um good at generating disinformation and lying even before this. And now you have the sweat shops. But LLMs do allow um bad actors to do this at an unprecedented scale. So think about for example uh fishing attacks. Imagine that uh every single uh voter in America gets a highly personalized series of emails, right? Trying to persuade them to take a particular form of action, right? So that's not something you can do with a sweat shop and unfortunately that's something you can do with with these AI systems. So, we do have the potential, this hasn't happened, to have kind of unprecedented attacks. One other thing, and I don't want to get into education because there's so much to say there, but I I do want to put a nail in the coffin of these uh LLMbased tools to detect LLM writing. And I'll make a

32:55 confession. Even though I consider myself to be a decent writer, I now use LLMs in my writing all the time. And the first thing I do after I use the LLM is I give it to the LLM and say, "Does this sound like an LLM and why?" And it often actually gives me subtle clues in the writing where it says, "Oh yeah, this aspect of it does sound like an LLM wrote it, right?" And I say, "Great, fix it."

33:22 So after a little, you know, I guess I'm a little bit more sophisticated than a seventh grader, you know, the night before the exam, but probably they're plenty sophisticated, too. So after a little round of uh write it, detect it, fix it, rewrite it, nobody can tell. And then I also have the audacity to make a few edits myself. Uh because I do come my writing and it's just a mixed thing that nobody can tell what Orin wrote and what the LLM wrote. And well, one thing we probably should add is there certainly are techniques where a company that provides an LLM can make that LLM such that you can detect uh its writing using these statistical watermarking technique. So the text seems just as fluent, but there are these uh mathematical uh clues uh embedded in terms of the distribution of certain letters and certain words.

34:19 And one can imagine regulations, you know, calling for for that. Um but the problem of course then will could be is that there'll be the unregulated LLMs running you know on the RU domains that uh people this uh will use instead.

34:45 Yeah. So one thing I just wanted to add um it builds on what Orin was saying I mean in terms of his own process right? So um I mean I assume he's not doing this in any malicious I mean sort of way. It's just I mean part of his process now right and so I mean thinking of the good actors that want to do the right thing and be transparent about it and so forth. So um uh we've actually created this AI attribution um sort of mark that um it's not just yes or no like did you or did you not uh use an AI to generate any text in this but it's very fine grained um so which uh of the two the human or the AI took the initiative um uh did the human review it um was it used just for polishing the grammar? Was it used for um uh generating like large sections? I mean all of those things and then it um we kind of have this thing very much like a creative common license. So it'll specify exactly like all the details of

35:43 what you did. Um and so then you can attach it to your writing to your publications whatever it is. Um and it can be a tool to to help people be more honest about uh about what they've done

35:56 or transparency about the attribution. Right. Right. Exactly. Exactly. So Kush there there are a lot of benchmarks to evaluate factuality or trustworthiness of an AI system. So can you help us understand what these bench benchmarks are actually evaluating if there is a general benchmark or one needs to use several of them to evaluate specific properties and if so then does it make sense to combine them into one single score for transportiness. So guide us through the space of these benchmarks. Sure. Um, so I mean as we've already gone through this conversation, we've talked about I mean all sorts of different dimensions um of trustworthiness of um even on the factuality side of hallucination side of it. I mean there's kind of factuality with respect to a kind of world knowledge or um it could be that you train the model only on um books that say that the earth is the center of the universe and then if it says that then

36:57 that isn't really hallucination or whatever but um uh I mean long outputs and the factuality in those versus short outputs. I mean there's all sorts of things that could be going on and um uh there's different benchmarks, different data sets that are out there um uh the basic way it works is just I mean you have some uh prompts or some inputs um and some expected outputs um and then you see does the LLM match those or not and um uh for example there's this thing called the agriact um benchmark collection it's like 11 different um uh different uh uh characterizations and so forth and uh all of these just build up. I mean that's just on hallucination then on hate speech on this on this on this um you keep putting it together and um you get these large collections of things um they might even take a very long time to run uh which is a bit crazy but um uh you can do some sort of model risk evaluation um there's even like one of

38:03 the IBM products called Watson extra governance I mean we'll do this for you there's other open- source things there's uh Helm from Stanford etc., etc. But at the end of the day, um that's just like one indicator. Um but you really need to see how these things are used in real life in like practical use cases um with um like very specific sort of needs that a that a given user might have. And that's not the way benchmarking is. It's not benchmarking is not going to reveal that, right? So it's uh it's something but it's not the whole story.

38:40 Henry and Orin, do you want to say more about benchmarking? But this goes takes us to the next question which is the contextuality of trust. No, because uh also in our community survey some respondents said that transportiness is contextual since it depends on the domain and the objectives that you have in using the an AI system in that specific scenario. So does it mean that an AI system cannot be trusted in general but its trustworthiness should be only evaluated when we know when and how it's going to be used? What do you think Orurin?

39:22 Um I I'm not sure that I that I have something uh worthwhile to say on this topic that's that's not obvious because obviously right everything is is contextual. I I do want to take this opportunity though just to mention some really interesting research that's related not exactly on target but I hope you'll humor me won the uh best paper award in uh uh Nurips just recently right 2025 on a on a benchmark where they took um open-ended questions you know questions like what do you think about abortion right where of course the answer is contextualized by so many things right what is a liberal think?

40:04 What does a conservative think? Uh uh what do you think about first term blah blah blah. But the thing that's really interesting there uh which is a related point is the homogenization. So it turned out surprisingly perhaps that whereas human responses on a question like this are incredibly diverse, right? You know you say like two people three opinions the responses across model and of course within model were remarkably homogeneous which is in its own way right non-rustworthy right it's kind of like regression to the mean of a certain sort mean opinion so I think it's just uh important to understand just how

40:49 unreal and you know optimized a mathematical function what you're getting is uh it seems like a genuine opinion. Uh what it is is right very much, you know, stoastic parrot to use that phrase.

41:06 Anybody wants to add to that? Uh um the homogeneity of the behavior of these systems or the content that generated. Well, I just say that that um you know that that is it it's it's one of those insights that's really important that in retrospect we say oh yeah of course because it's based on on merging so many human models uh in in in into one and particularly the uh and then after that you have this post-raining and most of the systems are all doing uh the same postraining I' I'd be curious I have to go read this paper now to see if they also tested Grock to see if uh if you take a lot of the guide uh rails off do you get some more um diversity in responses

42:03 but would fine tuning you just get what what Elon Musk thinks right it's tuned to sorry Francisca please go ahead

42:12 that's fine no I was saying that but um isn't finetuning or these postraining techniques Aren't they able to contextualize no a bit what the LLM is going to say for the specific task or the specific company or the specific community? So doesn't that help?

42:35 I'm not so sure about that. Right? Because um a lot of the reinforcement learning from human feedback which is the basis of a lot of this postraining um the humans are not expressing their own views. They're expressing what they're told to express. They're given like a bunch of guidelines from the company that hired them or the multiple layers of companies that hired them eventually, but um if they don't follow that, they don't get paid, right? And so um uh it's not their humanness that's coming through. it's just them as um uh some like other kind of computer really um at the end of the day and it's expressing the the policies or the uh the values or the desires of the the company that hired them. So I I don't think it actually happens um in the way that you're saying.

43:24 Okay. So um so let's go to now the the the topic of value alignment which is related and very linked to trustworthiness because in some sense I tend to trust more systems or people that I know they have values aligned to me. But how can we be aware of the values that an AI system operates with and to check if they're aligned with ours and what does it mean exactly to be aligned with human values? Yeah, I mean um this is exactly what Orin was just saying, right? I mean, the that paper, I haven't read it either, but um presumably it's um putting in these different kinds of prompts with uh different kinds of potential responses based on the underlying values um that would mediate those responses. And uh uh that's clearly I mean a way to do it. Um but uh I think there's kind of a need to go even beyond that because um uh like like if I'm a member of some community with some values like first of all how

44:31 do you even define what the bounding box of that uh community is in which there should be some homogeneity of values and um I think that's an open question. So like I mean we can talk about all these different sort of um like groupings like the conservative liberal or anything but maybe it's that uh apple farmers in upstate New York have a very specific sort of um uh value judgment on a particular type of uh query or response or whatever right so like I mean how do we get to to that point is the question and then uh like often times a person doesn't even know their values right I on most things like I don't know what I think. I mean I've never even thought about it. So um uh like just expecting people to be able to probe models to uh in the way that reveals their biases is a bit uh I mean too much sometimes. So um I think more like ethnographic sort of approaches let people use these things in the context that they want to

45:36 use them to try to satisfy their needs and have an outside observer kind of see where it's falling short might be a a way to go.

45:45 Okay. Any other thoughts about value alignment? [clears throat] So now let's go to you know now first we said the definition the history then the assessment or testing but now how do we can we be a little bit more proactive? So how can we mitigate the lack of actuality and trustworthiness? So so I ask orin again but I think that in your previous answer there was already the answer to this. So are there methods to improve the factuality of an image, a video or sound and so on. But if we can't even assess then uh can we how can we improve it? Is there some some solution or preliminary you know whatever thoughts of a solution?

46:38 Let let me answer this question in the context of video image type of content etc because these are kind of different species. I've for quite a while now certainly since the article I wrote back in 2019 uh advocated that we need to make more use of authentication and digital signatures. So these technologies uh already exist [clears throat] and um we um uh we can use them to sign and in that way authenticate right where this comes from. This really is a CNN video. this really is, you know, or it's only playing with his kids or or what have you. The problem here, interestingly, is not a technology problem and not an AI problem, but I think an adoption and user interface problem. I don't think it's a regulation problem uh either because I I um I don't think we can uh regulate our way to this and as Henry pointed out, right? Um you know, people can always use tools that that don't do this, right? If we regulate that say

47:44 open AI signs everything as they already do right generated by Sora uh that's not what's going to help us where I think we need is people need to be educated about the fact that if you see something on your f favorite social media network or YouTube what have you and it's not signed then there is no basis for knowing that it's real.

48:10 Yeah. And the problem is that we don't have that mindset uh today and there's a big question of how do we fix it? But I think that is the direction as opposed to you know building increasingly sophisticated detection tools etc.

48:26 Okay. But then I have a naive qu follow-up question which is are these digital signature robust? I mean can't I fake a digital signature or remove a digital signature and put another one. So are they robust? It

48:41 it's it's not a naive question at all. Uh and it has two pieces. The first one is that the uh signatures, you know, these are cryptographic protocols. They really are robust. Again, modular quantum computing and things like that. So when it's signed, you can rely on it. When it's not signed,

49:00 right, there's a funny thing. It could still be genuine even if it's not signed.

49:04 Yeah. But if it is signed, you can rely on it. So you can insi uh ins if you insist and again you know maybe it's the tools right your browser your app that you're using needs to insist on things being signed and if they're not signed there should at least be a very uh broad articulation hey this is unsigned content then I think we could be in good shape.

49:27 Uh unfortunately that's not how things are playing out right now. Yeah, but how would you sign a fully human generated uh piece of work, right? Like that I mean you can sign the fully AI generated thing, but uh isn't it more important to maybe prove that yes, this was fully done by me? Um

49:48 yeah, well you you would sign it. Let me give a very concrete example that illustrates how this is today. Today we get emails all the time, right? We exchanged some emails uh before this uh panel and we assumed that those emails were in fact generated by the people that said, but they're really easy to spoof. Somebody could have easily inserted an incorrect email. It's also really easy to turn on digital signatures. We just don't bother. So Kush, the emails that I got from you were unverified and I'm sorry to say I trusted them. Not because you're obviously a trustworthy person, but because I really shouldn't have. What I should have done is insisted that you digitally sign your email, which you can easily do, but none of us bother. So, so this is the situation. We we can easily trust human generated content if we just took a few extra steps. Those are not turned on by default. Uh, which I think

50:46 is a problem. Okay. So um Henry let's go on a different topic here because now uh many AI systems are given humanlike personalities. So is this something that goes in the direction of improving transportiness or not? In our community survey only 24% of the respondents agree that this d this is a good direction in terms of improving trustworthiness. What what do you think about it? Yeah, I I I think a lot of us are kind of freaked out at at that idea of of some sense tricking people. Uh uh Orin wrote a an oped years ago talking about how AI systems need to tell everybody upfront, oh, I'm an AI system. I'm not not a human. Um so I I I think that uh it's it's coming have having more of these these human personalities uh a certain percentage of users will trust such systems more more readily others will be more skeptical and there's certainly like an uncanny valley approach where uh if you take your AI

52:04 system and you put it in something that is clearly not human like a a a picture of an animated uh teddy bear or something. Um that comes off as being okay that I can now trust my teddy friend where a human would be kind of off-putting. But I think we're we're rapidly crossing that that uncanny valley. So I I don't think it does anything to help um it helps people trust the systems, you know, create this emotional sense of trust, but you know, we're already really good at doing that. We don't need any more encouragement.

52:45 Okay. So for the sake of time let's go to another question which is for Kush but also for everybody else that about the techniques that are being used to mitigate these issues with with factuality and transport for4iness such as fine-tuning reinforcement learning ra use of external tools uh chain of thought and many more. So um are these tools are these techniques effective um to verify there is also a question in the chat about the relationship between transportiness and verifiability. So can for example LLMs themselves like orin did you know generate but also verify and and check that what was generated is true. So because many of our survey respondents mentioned like a lot of techniques for um mitigating these issues like new neural network architecture but also external factchecking tools uh nobody mentioned because maybe we didn't give the the the option about using LLM themselves you

53:58 know to do like LLM as a judge to judge and verify the trustworthiness of others. So what can you tell us about this topic? Yeah, I mean there are tools there are I mean all the list I mean the whole list that you went through those are certainly things that one can use but none of this is perfect right and um just taking the chain of thought as an example um that I mean is like you just asked the LLM tell me how you got the answer right and it tells you something but that itself can also be hallucinated so we can't really rely on it um uh rag you give another example I I mean what this is is it's doing a search first getting some small number of documents and then using an LLM to kind of summarize that or draw extract an answer from it. Um and there's still the LLM in the middle right I mean even if you narrow down the um uh the search space or the content sources or whatever the LM still can hallucinate. So maybe

54:57 you'll do better but I mean not perfect. So um I think there is a need for um things outside of the model itself. So um uh yeah I mean one thing we've been developing is um uh this model called the granite guardian which is able to do some level of hallucination detection some level of um arm detection and so forth like after the fact or on the the inputs and so forth um but even that isn't enough right I mean so uh you can kind of go even deeper I mean do some kind of external searches for factual stuff um and then do some probabistic inference based on what you get back with the response. Um uh and try to even do more with that. Um so something we call the fact reasoner but um at the end of the day nothing is perfect and uh I think we keep need to keep working on things keep keep making progress.

55:53 Okay. and and H Andry uh uh something to follow up what coach said because LLMs are increasingly be used within the so-called agentic AI systems now that include LLMs but also include other components that by their nature they're more transporty and factual like traditional databases symbolic reasoning systems and is this a way to increase the transworthiness of the overall system even if includes LLMs Yeah, absolutely. And I think we've really seen remarkable progress in just the last few months um with LLMs augmented with tools to improve their trustworthiness or their factuality in specific ways. So for example, when you're uh using uh chat GTP, you can if you're a paid user, you can go in and select create a GPT for me and then you specify some additional parameters. uh such as I wanted to rely upon a um uh a scientific uh uh uh scholarship database. I don't remember which one it uses. And that pretty much eliminates um

57:05 hallucinated citations. It still might have other kinds of of hallucinations, but boy, when you're doing uh you know your your reference review and you're trying to track down your citations, that's that's an enormous help. Um and I think then when we see uh software uh AI systems for example uh used in uh software development we're starting to see some integration of uh uh of tools for type inference and uh simple kinds of program verification uh that will are going that are you know improving uh quality less sort of hallucinated code. So, so we're definitely are making good progress.

57:51 Yeah. Kush and Orin about relationship of a gentic AI approach to address trustworthiness issues.

58:00 Yeah. I mean, there's starting to be this um phrase going around called guardian agents. Um so kind of like good guys um that are trying to find all of the the bad stuff. Um kind of like uh antibodies and and so forth. But um yeah, who knows? We'll we'll see. So I [clears throat] I want to make two uh two points. One is I do think as Henry said and uh I actually hadn't tried that capability. So it's interesting to get your uh assessment. I I do think this is work in progress that's getting uh uh increasingly uh improved right as in practical terms.

58:40 It is interesting to pose a more theoretical question which is um how do we build the notion of factuality into the system? Is just a pragmatic matter of tuning it and designing architectures with multiple kind of critics and so on or um is there a way to build in the notion of a fact into the loss function or or into the system in a more robust way where you could get a guarantee. Right? I don't think Henry is claiming that you actually have a guarantee just as a practical matter. Uh it's uh it's working. That's one thing. The second thing is this problem in a funny way continues to expand. So a lot of the work that I've been doing in 2025 is going back to the old notion that Henry and I remember maybe others too of softbots and agents and so on. And now using this technology we're building agents. Now the problem of trustworthiness is do you trust your agent to operate on your desktop? Do you

59:36 trust your agent to send an email? Do you trust your agent to use your credit card? So, in a funny way, right, this problem even as we're resolving the hallucination problem is actually expanding, right? Trustworthy AI no longer means do you trust the kind of inert document that it generated is it's do you trust it to drive your car? Do you trust it to use your credit card?

1:00:00 Yeah. To act in the real world, you know. Exactly. As delegation. Okay. So I think we are already uh out of time but let's do one last round for everybody. So and the general question is are we optimistic or not? So will we be able to have factual and trustworthy AI system soon? 60% of our respondents said that they don't think so. uh so uh I would like to get your opinion and if you think we will be able to do it will we do it just with technical solutions or we will need also social technical ones including as we mentioned people awareness training uh you know and different frame of mind so let's start with uh Emry

1:00:46 yeah so I I guess I'd say I'm optimistic I think that within uh two years maybe less uh will really have the factuality problem solved and um uh for for most uses uh they'll be pretty darn uh trustworthy. Um and so that's that's so on the pro side.

1:01:11 Okay, good. Odin. Well, you know, I feel like our our panel has been woefully devoid of controversy. So, I will uh end on a note of

1:01:24 the controversy. I'm I'm uh quite pessimistic and I I I think that the fundamental problems uh I do agree with Henry that we're going to make excellent technical progress but I think the fundamental problem is a sociotechnical problem and we're really uh struggling uh with with those as a society. But would you say Orin that factuality is more of a technical problem and trustworthiness is more includes more of social technical things?

1:01:55 Yeah, I think that's a fair way to put it. Absolutely. Okay, Kush, let me not even answer the question to end on that note. Um, so I'll say that maybe um like asking for trustworthiness is maybe the the wrong question. Um because I mean circling back to what Henry said at the very beginning, I mean eventually it's a trust question, not a trustworthiness question. And um trust involves vulnerability and uh a lot of the people that are now going to be onboarded into using these AI systems are already vulnerable themselves for various reasons. Um they're uh in terms of their soio economic status or other marginalized reasons and so forth. And so asking them to trust something when they're already vulnerable is very problematic. So uh actually I will answer the question. So it is social technical and it's um really a question of um kind of doing things beyond just the AI beyond the technology and kind of

1:02:55 making people less vulnerable in other ways so that um uh the the AI can then become trusted in in their lives.

1:03:03 Okay. Thank you. Well, I mean not really controversy but three different angles to to to this uh question. So let me thank at this point the panelists for a very insightful discussion that I hope shed some light or given everybody some food for thoughts on AI factuality and transportiness. Thanks also to all of the people who listen to this and watch this panel for joining us. The panel has been recorded. The recording will be available soon on the triple AI website for the report on the future of AI research where you can also find already the previous file panels and on the triple AI YouTube channel. So I'll see you all soon for another interesting discussion on AI. Bye-bye.

1:03:51 Thank you.

Questions about this episode

Who speaks in "AAAI Presidential Panel on Factuality And Trustworthiness"?

Franchesca Karosi (Host).

What is discussed in "AAAI Presidential Panel on Factuality And Trustworthiness"?

In order: 0:00 Triple AI, IBM, Franchesca Karosi, AGI; 20:00 Joe Biden, 2025, New Hampshire primary, NIST; 30:00 AI attribution, Watson; 50:00 Granite Guardian; 1:00:00 Triple AI.

What is mentioned in this AAAI episode?

16 people, works and ideas are mentioned, including Franchesca Karosi, Triple AI, IBM, AGI, ChatGPT, AI perception versus reality. Each one on this page links to its own page with every episode that mentions it.

How long is "AAAI Presidential Panel on Factuality And Trustworthiness"?

The episode runs 1 h 4 min; the transcript is about 9,935 words in 96 timestamped sections.

More from AAAI

• All analyzed episodes
• People mentioned on AAAI
• Topics mentioned on AAAI