Transcript

AAAI Presidential Panel on AI Reasoning

From AAAI · Episode page · Watch on YouTube

· 1 h 8 min · 11,362 words · 88 timestamped sections

This is the full, timestamped transcript of "AAAI Presidential Panel on AI Reasoning", an episode of AAAI published August 7, 2025 (1 h 8 min, about 11,362 words). It mentions 13 people, works and ideas, including AI reasoning, Franchesc Rosi, Large language model, Subaro Raalo, Triple AI.

Who speaks in this episode?

What is discussed, and when

  1. – 10:00: Triple AI, AI reasoning, Large language model, Subaro Raalo
  2. – 20:00: AI reasoning, Large language model, SAT, SMT
  3. – 30:00: AI reasoning
  4. – 50:00: Large language model, Triple AI
  5. – 1:08:00: AI reasoning, Triple AI, AGI

Notable quotes from "AAAI Presidential Panel on AI Reasoning"

  • AIS global leader and I'm also the past president of triple AI. And I would like to welcome you to the triple AI panel discussion on AI reasoning. Before we get to the…
    about Franchesc Rosi · at
  • And safe. So to contribute to this goal in March 2025 during the triple AI conference, triple AI published a report on the future of AI research which summarizes the finding of…
    about Triple AI · at
  • Community survey reports about this topic. to follow up on this report, we already held two discussions on two of the 17 top topics. One on AGI and one on AI perception versus…
    about AGI · at
  • Which have played and continue to play a key role in critical real world application. More recently with large pre-trained systems like LLMs, they have made impressive…
    about AI reasoning · at
  • He's a professor of computer science at Arizona State University.
    about Subaro Raalo · at

Who and what is mentioned in this episode?

People

  • Franchesc Rosi — Person · Main speaker (host/moderator) · 1 mention · first at
    Opens the panel, frames the AAAI report context, introduces panelists, and moderates the discussion on AI reasoning and verification.
  • Subaro Raalo — Person · Panelist (referred to as 'Rao/Ralph') · 2 mentions · first at
    Explains reasoning as inference from base facts, contrasts LLMs vs 'reasoning models', discusses verification/generate-and-test, and critiques benchmark leakage and algorithmic generalization limits.
  • Alexander Bonholber — Person · Panelist (referenced as 'Olar/Olgar') · 1 mention · first at
    Provides a perspective emphasizing formal reasoning, guarantees, and the role of SAT/SMT in hardware verification; argues against equating LLM plausibility with sound reasoning.
    Is a fellow of triple AI urai ACM and is an expert in machine learning automated reasoning and optimization.

Companies & orgs

  • Triple AI — Company · Organizer · 7 mentions · first at
    Organization hosting the panel discussion and publishing reports on AI research trends.
  • IBM Research — Company · Affiliation · 1 mention · first at
    Employer of the panel moderator, Francesca Rossi.
    Hi everybody. I'm Franchesc Rosi. I work at IBM Research. I'm the company AIS global leader and I'm also the past president of triple AI. And I would like to welcome you to the triple AI panel.

Concepts & topics

  • AI reasoning — Concept · Core topic · 11 mentions · first at
    Central theme of the panel discussion, focusing on the differences between sound and plausible reasoning in AI systems.
  • Large language model — Concept · Key technology · 4 mentions · first at
    Discussed as a modern approach to AI that provides plausible reasoning capabilities.
    Which have played and continue to play a key role in critical real world application. More recently with large pre-trained systems like LLMs, they have made impressive…
  • SAT — Concept · Technology · 1 mention · first at
    Mentioned as a form of sound reasoning technique used in AI.
    There is also a critical need for verifiable and sound reasoning in AI systems. Over the decades, AI research has led to a range of automated reasoning techniques that have…
  • SMT — Concept · Technology · 1 mention · first at
    Mentioned as a form of sound reasoning technique used in AI.
    There is also a critical need for verifiable and sound reasoning in AI systems. Over the decades, AI research has led to a range of automated reasoning techniques that have…
  • AGI — Concept · Topic · 1 mention · first at
    Mentioned as one of the topics covered in previous discussions organized by triple AI.
  • AI perception versus reality — Concept · Topic · 1 mention · first at
    Mentioned as one of the topics covered in previous discussions organized by triple AI.
    Community survey reports about this topic. to follow up on this report, we already held two discussions on two of the 17 top topics. One on AGI and one on AI perception versus…

Places & events

  • triple AI conference — Event · Event · 2 mentions · first at
    Mentioned as the event where triple AI published a report on the future of AI research.
    Research and the challenges still ahead of us to make AI more capable, reliable and safe. So to contribute to this goal in March 2025 during the triple AI conference, triple AI…
  • triple AI website — Place · Resource · 1 mention · first at
    Mentioned as the place where the report on the future of AI research can be found.
    Disciplines, policy makers, funding agencies, the media and the general public. you can find the report on the triple ai website and you will see that is as identified in total…

Full transcript with timestamps

Hi everybody. Um I'm Franchesc Rosi. I work at IBM Research. I'm the company AIS global leader and I'm also the past president of triple AI. And I would like to welcome you to the triple AI panel discussion on AI reasoning. Before we get to the discussion, let me say more generally what triple AI did around re AI research recently. So we observed that AI capabilities and research were undergoing and are still undergoing a fast and very significant transformation along many dimensions the topics the methods the research community members and the working environment. So it's very important to be able to clearly identify the trajectories of AI and AI research and the challenges still ahead of us to make AI more capable, reliable and safe. So to contribute to this goal in March 2025 during the triple AI conference, triple AI published a report on the future of AI research which summarizes the finding of a study of

several months on significant AI research trends. Uh the study was conducted by 25 prominent researchers and supported also by a community survey with about 500 respondents and it covered many of these trends and dimensions. who is doing AI research, what new topic in AI is AI research about, how AI research is conducted and also mentioned the geopolitical scenario uh glo that globally uh is uh enabling and supporting AI research. This report has been intentionally written in a nontechnical way because it's intended to reach other audiences not just AI researchers including experts of other disciplines, policy makers, funding agencies, the media and the general public. uh you can find the report on the triple ai website and you will see that is as identified in total 17 topics uh and it has a chapter on each one of them and each chapter sketches the history of this topic the current trends

the open challenges and also what the community survey reports about this topic. uh to follow up on this report, we already held two discussions on two of the 17 top topics. One on AGI and one on AI perception versus reality. And today we will discuss the topic of AI reasoning. Uh of course reasoning has always been seen as a core characteristic of human intelligence. uh is used to derive new information from given knowledge and this new information is correct when sound formal reasoning is used otherwise we can say is a plausible information since AI is al also mostly but also trying to imitate human intelligence it's no surprise that AI researchers have tried to embed reasoning human reasoning in AI systems however it's also important not just to mimic AI uh not just to mimic human intelligence but also we want the technology to be reliable. Therefore, there is also a critical need for

verifiable and sound reasoning in AI systems. Over the decades, AI research has led to a range of automated reasoning techniques that have given rise to AI algorithms and systems including SAT, SMT, constraint solvers, probabilistic graphical model. All of which have played and continue to play a key role in critical real world application. More recently with large pre-trained systems like LLMs, uh they have made impressive advancement in their reasoning capabilities. uh but the reasoning appears less sound correct or more like plausible and uh so more research is needed if we want to guarantee correctness and depths by the reasoning performed by by them and such guarantees are particularly important for high stakes decision scenarios and for autonomous AI agents. So uh let me introduce the panelist of today's discussions. Um one is Olarus. Olar is Alexander Bonholber professor for AI at

NWTH Aken University in Germany. He's also a professor of machine learning at Leiden University in the Netherlands. He's the president of the European AI association euro Euroai. He's the chair of the board of Ka Confederation of Laboratories for AI research in Europe. uh is a fellow of triple AI urai ACM and is an expert in machine learning automated reasoning and optimization. Uh the second panelist is Subaro Raalo for friends and everybody uh Kapati. He's a professor of computer science at Arizona State University.

His research is mostly around around planning and reasoning capabilities of large language models or large reasoning models. He's a fellow of triple AI ACM AAS and the former president of AAAI. Ralph is also the recipient of this year outstanding educator award and is very active on Twitter X and LinkedIn explaining AI concepts. So uh let's before we start with the questions to the panelist let me remind everybody to write your questions in chat. We will try to address them possibly towards the end of the panel discussion.

Uh so let's start with Ralph. So let's start from the basic here about reasoning because many AI papers and I'm sure the audience have seen them claim that AI is able to reason but many others claim that is not able to reason. So it seems to me that the reason for this discrepancy may be in the different definition of reasoning that the authors of the paper use. So let's start from there. what exactly is reasoning according to you and what role does it play in in human intelligence?

Yeah. So um in intro to AI we talk about the fact that in especially most formal systems uh you start with some base facts and you are able to find the inference from them and the the idea of finding you know doing inference uh to reach valid conclusions uh or plausible conclusions in some cases or probabilistic conclusions has been seen as reasoning. And the interesting part of course as as you have mentioned uh in the beginning itself um reasoning is what AI was doing from the beginning. You if you start from general problem solver GPS and all the things that New and Simon etc were doing they were essentially in the business of giving the base facts that the humans will write down and then focusing on can the AI systems do long um chains of inference in you know with some guarantees which is part of reason why we were able to actually have a chess machine winning over human um um grandma

know human champion Gary Kasparov back in um late 90s right what changed is those systems were sort of deep and narrow systems they would start with some knowledge that you have written you have acquired and written for the AI system and then the system itself focuses more on the inference part and you know starting like 2013 onwards we sort of gotten very much interested in these pre-trade large language models which essentially have been trained on the entire digital footprints of the humanity and and and so the question then is are they somehow going to be able to of course when you do train that they will be able to integrate pieces of information provide shallow inference on this broad knowledge they've been trained on the real question is are they actually able to actually do uh longchain reasoning the answer for that turned out to be that the normal LLMs couldn't but then if you actually

somehow also train them on the reasoning traces which is what has been called LRMS starting 01 and so on. um they can actually imitate some parts of the reasoning performance and and the way I'd like to say this is originally LLMs have been trained on the knowledge web that we wrote for each other and what now that we're doing is we kind of creating a synthetic web where with you know in huge number of problems on their solutions and then you're training the LLMs on those too and that's partly the way we are imitating reasoning in these kinds of systems.

Okay. Uh Olgar. Uh but before we go to AI reasoning, so tell me your opinion about what reasoning is or and then should be maybe when we try to imitate it.

It's it's a fascinating question, right? Um and I think it is one that even philosophers haven't fully answered. Um Rao already mentioned there's actually different kinds of reasoning. uh even in philosophy there's different kinds of reasonings that are being distinguished from each other uh such as inductive reasoning uh deductive reasoning in particular right um and so I think it's really important um to be aware of the fact that that reasoning is sort of a distinct um achievement of human culture right you can trace this back way before the period where any sort of um higher technology emerges you know for instance in ancient Greece Um, philosophers spend a lot of time thinking about correct modes of reasoning. And as as Raul said earlier, correct modes of reasoning are s are those that that basically from premises that you believe in for whatever reason because for example, you've observed

something in nature, right? Can give you um deductions, can give you further statements um that then turn out to be reliable and true, right? And now of course um whereas the ancient Greeks have mostly been concerned with a kind of deductive reasoning that that is very deterministic right. So you you assume that you have some facts you have these um basically rules of reasoning and you derive new true facts from them with certainty. Of course, later on it's been realized that there are other kinds of situations where you can't be entirely sure about the knowledge that you have, where you have to deal with uncertainty and so on, and much progress has been made over the centuries um to basically get to more general modes of reasoning.

And then of course there is what you might call common sense reasoning. It's it's basically what people use in order to deal with everyday situations without um without resorting to any sort of formally trained logic, right? And so what what I would like to say maybe a little bit provocatively is that this idea that you get good reasoning the kind of reasoning that as humans we use all the time for instance in engineering or in mathematics right um the first thing to realize is that people aren't very good at this. It's very hard for people to learn this and therefore it's been a great cultural achievement to understand the basics of these kind of reasonings and we've been working on them for more than 2,000 years and we should be very proud of ourselves that we figured out basically as a human as human societies um to reason well enough that we can base technology on that and computers in particular wouldn't exist

if it weren't for that kind of I would call that logical formal reasoning with guarantees in some sense you can you can see computers as the embodiment of that sort of logical reasoning. Right? And so for that reason, it seems a little strange to me that nowadays many people with superficial knowledge of AI and even some people with very deep knowledge of AI as a field seem to be content in saying surely if we just train on enough data and we have a model that's complex enough, reasoning will somehow magically emerge just as it did with humans. I think it didn't magically emerge with humans. I think it was the effort of a very long and complex cultural process and any of us here I think that's Ralph that's you Francesca that that's myself and that's probably many in the audience as well who've had anything to do with teaching students to think logically right um know that it is not a natural thing it's something that

you have to study and learn and try very hard at before you get even reasonably good and then machines from the very beginning by design have always been better at it than we have not at the at the level of you know grandmastery in chess but at the level of just forming long chains of the kind of reasoning steps you need for instance for arithmetics and just to conclude um this little part um with that I mean you don't have to look at sort of deep logical reasonings to find um limitations of of current large language models and also large reasoning models you can go to simple arithmetics and you find that they can't do that very well because that too requires some sort of formal reasoning that that they're just not made for and that they're not good at.

Yeah. So to follow up on this question, um so tell us I mean especially for very young people in the AI research community that may not be aware of some success stories of AI reasoning in the past, you know, or of sound and correct reasoning embedded in AI systems. Um maybe you can tell us a little bit more about some of these success stories. After all um of the respondents of our survey I think that 79 almost 80% of them indicated that the topic of reasoning is relevant to their research. uh but maybe some of them are not aware of these success stories of sound research, sound reasoning that AI researchers produced and used to build computer and systems that we still use today. Can you say something about that Olar?

Sure. I'll try to keep it brief because I know that Rao has a lot to say about that since he's an expert on on many things including AI planning which which is one of the domains where there have been a lot of successes and continue to be. So let me just briefly mention something else. We wouldn't have any of the hardware that we currently use not just for AI. So that's GPU, CPUs and so on. Um but also for our mobile devices for pretty much you know everything that's driving um our banking system, you know, our public administration, uh the computers we use for research and engineering and so on, right? We wouldn't have any of this if it weren't for AI reasoning techniques such as SAT and SMT solvers because they're used in order to verify in many cases not completely but in key parts the hardware that we're using at tremendous scale.

So, you know, take Nvidia. Nvidia wouldn't be possible. Their products wouldn't be possible. The way we use their GPUs would be completely impossible without AI reasoning techniques being used to verify major parts of the chips that that they're producing. And that I would actually claim is the biggest success that AI has produced so far. the ability to build microchips, electronic circuitry of a complexity that no human or team of humans could handle handle possibly um you know at least not in a way such we that so that we can trust these things um and and still have that and use it and rely on that in critical applications if it weren't for these kind of AI reasoning techniques.

Thank you Ralph. Do you want to add anything about planning or anything? Yeah. Um I mean in in general um I can actually say that I mean the planning techniques have been used by NASA mission planning is mostly uh using uh probably correct planning techniques and so on model based reasoning but I also want to kind of mention the following thing that much of computer science really algorithmic reasoning is a a very strong form of reasoning. certain types of algorithmic reasoning has actually become the perview of what people called AI. But you know like SAT for example satisfiability is something that you know there are algorithms Davis partnner procedure is in fact one of the earliest JCM papers. The one thing that I want to mention is while it is important to think about um deep and narrow systems which AI was very good at before the onset of uh the LLMs and and broad and shallow stuff. Um it is also useful to

actually have this jack of all trades. Um which basically can in fact we'll talk more about it I'm sure as we go on. They can provide sort of reasoning shortcuts for somebody who is in the loop especially either a formal reasoner or humans in the loop. They can provide ideas and shortcuts which as long as somebody else can directly they can actually verify you know they're great idea generating machines. And so we may be able to find uses for these sort of shallow um plausible reasoning even if it's not guaranteed correct. My only worry would be when people say let's just put them in the end user facing safety critical scenarios and see what happens and that see what happens would not be very pretty because uh there would actually be without guarantees essentially there can be very costly failures.

Yeah. So let's go a little bit deeper into that. No. So because we said that we have these two forms of reasoning. One is sound and correct, one is just plausible. Symbolic AI has produced sound reasoning system. Modern LLMs can perform instead plausible reasoning. uh when we asked in the survey to the community um about properties that they would like to see in a in a process to be called the reasoning um 77% of them said that knowledge should be incorporated 72.5% that explanations should be provided and uh 57% said that it should involve multiple step to arrive at a conclusion but only 37.4% 4% said that there should be guaranteed correctness of inference results and outcomes and only 23.7% said that a formal system and solver should be used. So it seems to imply here uh that there is really a recent focus on informal plausible reasoning more than on sound reasoning. So let's go a little bit deeper about whether one

formal reasoning is better than another one or there is space for both in different scenarios. Um and and also that what after all you know this plausible reasoning is what human reasoning is and emerges from our brain. We don't have some reasoning rules written in our brain. So isn't that similar to what happens with LLMs? So what where do we use where should we use one or the other form of reasoning that now AI can provide? Do you want to start Ralph?

Yeah. So first of all as as you mentioned and I would actually say the following thing that even before the broad and shallow systems like LLMs uh we have had uh and looked at plausible reasoning. Analogical reasoning is a famously plausible reasoning. We realize that plausible reasoning is extremely important. Um, one of the ideas is I keep telling people that you should let your ideas come from anywhere. U but then you want to prove if in fact safety matters then the idea that you have you may actually have to check whether it's actually correct. So if you know kind of um limit yourself to only having correct ideas then you won't think much at all.

And that's sort of the part that was known before even within plausible reasoning in the AI uh and and cognitive science. And what is interesting is plausible reasoning and analogy is something that LLMs are actually pretty good at doing to some extent mostly because they've been trained on the entire collection of human knowledge at least as we have uploaded it to the web. We never had something like this. As you might know, some of us, some of you guys old enough to remember. Doug Lena tried to do this thing of let me write it by hand. And the problem is it becomes extremely hard. What happened with the web is everybody did whatever the heck they wanted in natural language. And we have these machines that do a reasonable shallow inference on these kinds of knowledge. The interesting part is and in fact even factual knowledge that LLMs produce is only plausible. That's where your huge hallucination points come

along. So they can't even actually directly with guarantees answer factual reasoning. If you want factual reasoning, you can always go back to the boring um databases. They will never make up stuff. But on the other hand, there's a good thing to talk about how to combine all sorts of pieces of information. I'm pretty sure a lot of us are actually dependent to some extent um on these AI overviews when you do search because you are still in the loop and that AI overview might actually provide you interesting way of combining pieces.

The thing that is interesting is how do you go from plausible reasoning to reasoning that has soundness guarantees. Um there are some obvious ways. One idea is to generate and test. And as Holga was saying and we were talking about earlier, much of classical computer science and artificial intelligence has been focused on providing principled solvers and verifiers. And in fact those verifiers are extremely useful. uh you can actually have a SAT verifier that if you say here is a particular uh satisfiability assignment the verifier can check if in fact it's a correct assignment and you can let the LLMs come up with suggestions and then the verifier checks. This is a way in which you can actually easily provide correctness guarantees and one other thing that that has been actually called test time scaling in the current LLM literature. The other thing is as Minsky said um part of intelligence much of

intelligence is take the test part of generate test and put it in the generate part as much as possible and that's what has been called post training in LLMs. So in essence you kind of train the LLM with verifierbased reasoning uh during the training itself so that the kinds of suggestions it makes are more likely to be closer to correct but still there is no guarantees until unless again you have verifiers in the loop. The the very interesting thing this brings up in essence is LLMs are amazingly good at capturing the style of anything that they have been trained on and that's something that we have seen that things like Shakespearean amic pentameter is so easy for them and it's so darn hard for us and so one of the interesting questions was if you show them enough algorithms and after all you know human web the web that we created does have algorithms Wikipedia has algorithms can LLM can just do internally simulate

algorithms. The answer is no, they couldn't. Which is actually why the standard LLMs basically for which there were lots of claims that they can do reasoning, they they kind of got replaced by these quote unquote reasoning models. If in fact the original ones are doing reasoning, then you don't need reasoning models. And so every even the people who say that they can do reasoning admit that they weren't doing reasoning. And the way now they are doing reasoning now is not you can't just read a an algorithm and you know execute it. Instead, you use the algorithm to or a verifier to actually generate uh multiple traces and see which of them are correct and train the LLM on that. And that is a very inefficient way of training the reasoning in but it is something that you are not paying for because somebody else with large amounts of uh um you know reasoning infrastructure GPU infrastructure is doing it and for you

the end user right now it's not costly and so it will actually pro improve the plausibility of the inference but without guarantees still

thanks Ol do you want to comment on this use of the two form of reasoning whether they're useful that one is better than another one and the relationship with this idea that plausible reasoning like in LLM emerges just like it emerges from our brain. So I think it's possible that plausible reasoning, you know, the shallow form uh with no guarantees, uristic reasoning emerges, right? Um that that's about as plausible as, you know, kids picking it up without any formal instruction ever. And and I think there's some evidence that that they do, right? I mean, there are even some other species that without sort of any any form of cultural transmission um apparently can can, you know, master tasks that require uh a few steps of reasoning. Um but but that's actually not the question. The question is you know how deep and how reliable does the reasoning have to be in order to do certain important things. And and

here I think it's very important to realize that that one of the ways in which generative AI LLM's large reasoning models are being used increasingly as we speak is for generating computer code. Right? I'm not even talking about inventing new algorithms. I'm talking about coding and software engineering tasks related to that and in fact um you know it has been argued that that this is one of the ways in which uh this technology brings the most immediate value um and of course you know we all know that it's difficult to to train good programmers and there's going to be there is already a shortage and that shortage is going to get worse.

Um so therefore I think it's very important to talk about this elephant in the room. Can this technology LLM's large reasoning models be good enough at the kind of reasoning you need in order to use them for coding? And you know, actually we're doing a bit of research uh on this uh at at my institute. Uh I think RA has looked pretty deeply into that as well and and so have many others. And I guess what we what we see is is not surprising at all. it is you need actually pretty deep and sophisticated reasoning with guarantees in order to make uh correct software.

Software that's free of vulnerabilities um and software that's reasonably efficient is then of course even harder, right? And so the the the question is not so much can LLM help um expert coders be more efficient although there's even some uncertainty about that. You might have seen there's contradictory recent studies on on that topic. Um but the real question is can that be used in order to have non-expert coders replace what currently expert coders are used for and I think the answer is if you just look at the evidence a very much resounding no yesterday um there was a major security issue coming from Microsoft actually from SharePoint I I think many of you will have heard about that uh that is something deeply concerning and of course we we have no idea whether that was a result of any AI coding techniques being But what I would say and what I would speculate in an educated way is if

we continue to go down that path where we ignore the requirements of you know relatively deep and sophisticated logical and mathematical thinking um when generating computer code and software we will have a lot more problems like this in the near future and that I think is a great concern. I think that could in fact sink the entire AI boom that we currently have because just imagine a world in which computer systems become even less reliable than they tend to be even nowadays sometimes, right? I don't think that that would be particularly good. In fact, a lot of what we do at universities and and other places that that train people who build systems is of course to make them better at building systems that perform according to specs and that have no vulnerability. So my feeling is that we should what we should spend effort on um is actually making these people more effective in what they do and I'm not

sure we're doing it um sort of blasting down the path that's currently being uh laid out in front of us. Uh believing somewhat blindly and without good evidence. I would say that you know um the kind of formal reasoning that's needed in order to have success there somehow magically emerges along the way if we just you know train these things more smartly. I think that's pretty naive.

Yeah. So, you also touched upon the collaboration between humans and machines that we will elaborate more later on. But now, let's uh also focus on there are a lot of benchmarks that try to check whether these AI systems um are reasoning or not. And if you pass the test, then you know researchers say that they're reasoning otherwise they're not. So how should AI reasoning actually be tested? Are the current benchmarks good enough? Should we think of new ones? And if there are new ones, what properties should be these benchmarks be to be good benchmark tests for AI reasoning? Ralph, do you have thoughts about that?

Yeah. So before I get into the benchmarks, uh let me just respond to one thing that Holar said and I would do without contradicting anything that Holar said, uh let me just say that the progress is not that bad. Honestly, it's it's actually the ignorance that might be worrisome to holar that people who might be thinking that LLMs and LRM have properties that they don't. Uh but they themselves are magical and useful for humanity right now. So if you think in terms of you know retrieval on one end and reasoning on the other hand LLMs come to like approximate retrieval LRM kind of come to approximate reasoning and it's a useful advantage to have these highly broad but you know no guarantee systems um for lots and lots of applications u when again if you put them right in front of safety critical or enduserfacing applications you get what you deserve And there is no reason to feel sorry for you at that point. But

on the other hand, I do actually think that getting broad and shallow models to be little deeper is a very useful research direction and that is indeed what is happening in the LRM right now. uh again no research is ever uh like it know goes in one single step and so there are obviously you know all sorts of mistakes that are currently happening for example that these reasoning uh systems LRM tend to provide intermediate tokens before giving the solution and there has been a mythos that developed that these intermediate tokens are thoughts that uh LLM is entertaining and there's actually no real reason to believe that whatsoever uh it turns out that we of you know work for example there is a paper from a group on beyond semantics that people can look at that um as well as a position paper that kind of argues that the intermediate tokens are helpful to the LLMs but they don't necessarily have to be helpful to the

humans at all and making trying to make sense of these intermediate tokens is like looking at roshock test inkplot test and saying oh there is an elephant there there's a parrot there you are seeing it they are not there so that's the first point coming back to benchmark this has been a very interesting problem about LLMs in general. Now, machine learning has always known that if you train a system on a particular distribution, when you are testing, you should make sure that the test data is from the same distribution or out of distribution, but not exactly the data that you trained on because that would be just memorization. But when you are training these systems that are being trained on the entire universe of data that is available, there is no such thing as out of distribution. What is today's benchmark becomes tomorrow's training data. This has become very much clear with things like MMLU etc. which

started as reasonable benchmarks and now everybody knows that the systems have been trained on those data whether it's deliberately or by actually accident. And so in in general, one thing that people really need to understand about LLMs and LRM is their capabilities come are are basically memory effects. the stuff that they have been trained on and we have extremely it's very hard to completely characterize what things the LLM have seen what things they have not seen because of which this famous uh thing about jagged intelligence comes that they can for example solve something like integration by parts and fail on a multiplication problem while you'll never see a human that can do uh the part integration by parts and fail on multiplication It's because what is present in the memory. So we actually did a recent study that shows for example that if you train a transformer on complex mazes and it does well on

that sort of complex mazes and then you ask it to solve problems where there is no maze that means it's actually free space start and goal is right in front of each other. It dies and in fact it has nothing to do with the computational complexity. LLMs and LRM don't solve problems in the classical computational complexity way. They're solving in terms of the distributional distance between what they have and what they're trying to solve. And in in general because we did this for like carefully constructed you know training and testing but normal LLMs right now essentially everything has been seen by them. In fact yesterday we talked about IMO uh stuff and so on.

There is huge amounts of IMO data. International Mathematics Olympia data that's been fed into LLMs and so they are a kind of a memory effects and I tend to say that LLMs are good at humanity's knowledge closure. So if you think of entire humanity they can do shallow closure of that knowledge but not like you know basically the reasoning parts can get you there and then basically the farther it goes from the closure uh the farther it goes from the humanities current knowledge the harder they are the the worse their performance is and so yeah go ahead

now before I give it to Olar I wanted to comment on what you said because it's not really the entire humanity knowledge is what is represented on the web

which for some cultures is much less compared to other ones. So there are certainly it is but

for entire parts of the world where the connectivity for example is much much less. So of course there is less representation on the web

except that it's by comparison to you know pretrain model thing this is way closer to the entire humanities knowledge. Yeah, but we have to be aware that not everybody is equally represented on this uh knowledge on which the LLM build the closure. Olar, go ahead.

And and furthermore, I think we have to be aware that that really if you truly train something on everything that's out there, it's likely that you get inconsistencies of course, right? I I don't think what you call the the the the knowledge closure of humanity row is in fact so desirable in the sense that from a purely logical mechanistic point of view it's likely going to be inconsistent and full of errors right and of course you you train on that too um however I I think there's one point that you made that I really want to just emphasize a little bit more and and that is you know because current um generative AI technology to a large degree at the bleeding edge is proprietary and we don't know really precisely what data it's been trained on and what training procedures have been used and so on. Right? It's actually quite difficult even for somebody who has the vest in attention and lots lots

of resources um to explore the limits of this. Right? It's it's basically it's it's a lot of black and blackish gray box uh evaluation of these systems and I really think it it shouldn't have to be that way right this is the way we test human knowledge as well but let's be very honest what we don't know what we don't need actually is sort of more systems that have all the problems that that people have in the work they do. It's not really it shouldn't be about replacing people in in all sorts of activities that they may be good or not so good at. It should really be about helping people deal with problems that perhaps they are not so good at efficiently or help them, you know, do more of the stuff that they than they could than they could do without the systems without replicating all the problems that that typical humans would have, right? And we know perfectly well that students who for instance um cramm

for exams um will typically not arrive at deep understanding and will display some of the same sort of you know problematic patterns of of performance that that we see in in generative models, right? Uh and we also know what it usually takes to get students further. And we don't even have to talk about university students. the same applies even for you know high school students and and for for kids at at at lower levels of schooling. So my feeling is that absolutely the current technology is very useful. There's no question about that. But it's crucial that we spend more effort um at understanding the limitations and then being very open and transparent about the limitations and enabling those who are not AI experts and who can't do these sort of experiments and testing for themselves to make smart and responsible decisions about their use of the technology. And and Ralph was entirely correct in sus in suspecting

that this is what I'm really worried about because I can see this happening right now. not so smart choices being made um because of you know a combination of of marketing by certain players and of course that's you know it's understandable that companies want to sell their products that's perfectly fine actually right but then there also not being enough sort of unbiased um views on on what this stuff is actually good for and I do think there we we need we need a bit of a correction and reasoning is one of the areas where the you know where it's really problematic and and it's problematic because reasoning is what you need in order to get um guarantees where they matter. And of course, Ral said correctly, not everywhere guarantees are crucial, right? But there are many places in society where we need guarantees. Um and it's not just safety critical stuff like you know um running parts of our banking

system or or running nuclear power plants or anything like that. It's it's much more mundane and smaller things. I mean think of the software that is being used for instance um by the tax authorities. what a disaster it would be if that software were, you know, of lesser quality than we have today for everyone, right? And so I think we need to get smarter in understanding where guarantees matter. And and just as a final thought on this, there are many many areas where machine learning technology is used. For example, uh in the medical field where robustness matters greatly, right? And reasoning techniques are also absolutely key in ascertaining robustness of machine learning techniques. And these reasoning techniques that are being used today are thankfully not really large reasoning model based or large language model based because that wouldn't be sufficient at all. Right? So they are of

the different flavor that AI thankfully also has in stock. And in fact if you combine these things together you get already significantly added value today right in the form of more robust machine learning models. So that is another direction that I think is extremely important for people to understand um can bring value to the entire enterprise and can make the machine learning methods that we use actually more responsibly applicable in certain areas such as medicine, finance and engineering and so on.

Yeah. So you you touched upon this and both of you about the relationship between human and LLM kind of reasoning and and other AI kind of reasoning because especially with LLMs as you know there is also this personification this anthropomorphization of LLMs. You know LLMs write like a human being. So people tend to think that they also have other characteristics of human beings such as you know reasoning also and logic reasoning and so that's really can create some confusion about what an AI system can provide and maybe it's not what it can provide because it has a different form of reasoning. uh even even knowing that as a in the in our survey for in our community um about almost 42% of people that says that focusing reasoning research in AI on human level reasoning is valuable or 47% they say even essential so is that really essential according to you that that AI can try to simulate human

reasoning or it should just provide a different form of reasoning and we should not try to im imitate and emulate human reasoning you

maybe I'll start quickly maybe I'll start quickly because it directly connects into what I said before so let me try even crisper I think it's a mistake to focus research on reasoning on trying to imitate human reasoning because human reasoning is quite weak and fallable right so what we've achieved over the past 5000 years is to actually develop techniques that are better than human re reasoning that are by leaps and bounds better and these ways have allowed us to build the technology that's underlying all of AI and all of computer science and all of its applications today. So therefore merely merely reaching human levels of reasoning would be a very low bar a problematically low bar. What we need to do is reach the levels of reasoning or actually just make sure that they're not forgotten about and that they applied when they needed that we already have and then maybe further improve on them.

And I don't think teaching large models how to reason is the best way to get there. But one can of course have a meaningful debate about that. Right? But the most important thing from my perspective is to realize it needs more than human reasoning to do a lot of things that we're already doing quite well that are essential. And that more than human reasoning doesn't come from generative AI nowadays. Nor is there any evidence that it will come from it in the near future. So let's use what we have. Let's try to use it in meaningful combinations with this other kind of AI.

Right? And of course that gets us to the point why should we have one model that does it all? Why don't we plug things together so that you know the right tool is used for the right job?

Yeah, Ralph. Yeah. So, I I agree with the part that if civilization depended only on individual human reasoning, not about the scale effects of human civilization, we would still be in beautifully decorated caves, uh not having these zoom conversations, etc., etc. I mean humanity's human civilization is way has a much bigger scale effect than what individual human that you can pick off the street might or might not be able to do. The interesting point here is people actually confuse this from even the um you know more LLM supportive uh side. uh they both actually say when for example you say LLMs don't do multiplication that and have you seen humans which can who can do 20 by 20digit multiplication and also take credit for the fact that LLMs and LRM can do well in international mathematics olympiad which about seven or 20 people actually even understand those questions and that's not really very humanlike I

would say in terms of normal humanlike in general What we really want is support for reasoning of all kinds and then with you know guarantees. I once again still think that it is extremely useful that this technology exists because we never before had this ability to bring to bear tons and tons of you know uh knowledge that's all over the place and and and then basically see what sort of shallow what you can get from shallow inference or slightly increasing length inference uh from that sort of knowledge and and so they can actually force multiply um you know our productivity in multiple ways. The most important thing and I hope this particular panel and other panels like this will help is understanding having healthy skepticism rather than be overly optimistic and overly pessimistic about these models. I mean remember you know AI used to be a thing that only people that will go to either triple AI or new rips would talk

about. right now your grandmother and your grandfather know more about AI than you know in terms of at least from the news perspective and so it sort of becomes it also comes with the territory with this great success we have that there is significant misunderstandings about what the c you know um the technology is able to do and and it's sort of very useful to understand those distinctions and I do want to point out one crisp distinction that the current LRMS are actually still uh um you know still have that limitation. We have shown this as soon as 01 came and there's a TMLR paper on that which is think of something extremely simple.

Don't talk about planning. Don't talk about sad nothing. Think of this thing called last letter concatenation. If I give you three words, then you are supposed to take the last letter of each of the words and make it into a string. You understood this. Now, now if I again give you another three words, you can answer this. If I give you 10 words, you can still answer this. If I give you 25 words, you can still answer that. What we see with LRM and of course also in LLMs is that they will do well at the place where you gave the size of the last letter concatenation examples you gave. If you gave like three four letter examples, they'll do well at three fourletter examples. They will die as it becomes 10 20. What that tells you and this has been seen across the board in planning and reasoning problems. What that tells you is that they don't learn algorithms directly. They learn unrollings of the algorithms. So they

are basically I'm good for 4x4 last letter concatenation problems. I am good for 3x3digit multiplications. This we we laugh at because we'll say that kids are supposed to understand how to do multiplication and follow that procedure how boring it is. But LLMs are not actually good at following procedures. They're not learning algorithms. They're learning unrolling of these algorithms and it's a very useful thing because the reason they do that is because that makes it easier to combine that into the current training regime of LLMs. Each of these unrolling things if you look at for example deepc car 1 it tries to solve it starts with a base model that has what is called top 15 accuracy that is reasonable in the sense if it makes 15 guesses with 90% success one of them is the correct answer and then it gives itself new problems tries to guess their solutions the verifier checks which ones

are correct and this laborious RL process tries to improve this base model when you do this it actually has a slightly better top 15 accuracy and larger problems but then it becomes again zero when you go further about that and so you have to retrain it again. This is something that actually people have to understand that they're not learning algorithms, they're learning uh unrollings of the algorithms because most of this is a computer science crowd. They will understand that distinction.

And by the way, this is what actually got repeated by this Apple uh study. You know we wrote this actually back in like know TMLR has the paper but we seep but they actually wind up showing the same thing that as you increase the length of the blocks world planning problems it will finally die you know etc etc because basically they are unable to follow any procedure

all good yes and just to connect to that that is precisely why they can't solve non-trivial programming problems right because you need loops for that right and the loops better not be sort of depthbounded uh so so extremely And that's also why why they why they will not be particularly good at at doing science in the foreseeable future, right? Because that also needs more than just um you know learning from from lengthbounded examples. I'd like to come back to one point that that you made earlier val which I think is is interesting and insightful. Um and and that is this this remark about computational complexity right um and without going into any technical details here I mean we know and have reasons to believe computer scientists have worked on this for a very long time and excellent mathematicians as well that there are interesting problems in the world many that require great complexity

to solve in their generality. Right. So many of the ones that you mentioned now fall into that category. Many types of programming tasks fall into that category as well. And there's no reason whatsoever to believe right now that there should be major shortcuts to this. And it would of course be the case that you know even if very very large um reasoning models um were were capable of of solving these problems in their generality, it would be the case that we would have basically shown that these problems are not nearly as complex as we think. And and I think from a from from for very deep theoretical reasons one should be doubtful of something like that. Doesn't mean that it can't be true but it means that the burden of proof the standard for proving something like this or for making people believe it should be very very high indeed. Um and and to me that also indicates that you know we we need to get even better than

we already are in in talking about the tasks that AI techniques that have been developed over the last you know 30 20 40 50 60 years and are being used very successfully in practice will not likely be simply replaced by at least the current kind of LLM and LRM technology. Um I think it's very important to understand these these sort of that that there are these deeper reasons even if not everybody can follow the deeper reasons but they do exist and I think even the trained computer scientists who make other claims would not go as far as saying this is now evidence that these problems aren't as complex as people have thought because I think people would stop taking them seriously. The real risk of course is also what Ralph said earlier that that that now where everybody talks about um you know AI and it's no longer just um subject of of of expert conversations we have very influential people like you know Obama

is one of them who start making claims about the capabilities of AI systems and people believe these claims simply because somebody who you know they respect and and trust in many ways has made the claim right so it's very un it's very important in in this sort of situation to understand that there's a difference when experts that that you know are really trained computer scientists and trained AI people make claims and and somebody who might be very famous or might be you know a CEO of a big company makes claims there is a difference there right the difference is not that one is surely right and the other is surely wrong but the difference is that that one of them has more of the background it one needs to even know whether something is right or wrong whereas the other has to has to rely on secondhand information.

Okay. So I think that we have uh analyze many dimensions or or sorry Ral do you want to say something else?

I just wanted to add one thing. One of the nice things about doing this for triplei is we have the uh great people in the audience. Ron Bman pointed out something that I've been saying separately. Uh in when you have the web, you can have explicit or declarative knowledge or procedural knowledge and LLMs are actually not the unrolling aspect can be thought of in terms of they're unable to unroll the procedures. They're not actually learning the procedure directly. And so that is part of the current limitation of LRM and that's what you laboriously use RL post training to increase the inference horizon. So in a way the signal in the verifier which actually checks whether the plan is correct or not that for any length of the plan is slowly being compiled into the generator in this incremental um you know horizon extension which could be fine if in fact you never want to do more than let's say

100 by 100 um you know last letter 100 by 100 word last letter concatenation problems but one of the things is as we know in computer science the final answer might well wind up having some very repetitive computation lowered down and if you fail there you'll get the wrong answer here even if in fact this doesn't have those many rulings in general it's too early to give up on all your computer science background keep that and understand LLMs LRM which I think is what much of this panel is really saying

yeah so uh I actually have a follow-up question on what you said Raal and mentioning also what Ron wrote in the chat about explicit and implic implicit knowledge because at some point I thought that symbolic AI was more about the handling of explicit knowledge and um machine learning and LLM were more about implicit knowledge uh because they were not symbolic and also you know we discussed at length you know the system one system two analogy with the the the thinking fast and slow theory of human decision making where the system one is more like implicit knowledge and system two is more like explicit knowledge and usually system one is more photo to be similar to LLMs while system two is more similar to a symbolic AI system. So how do you reconcile this a observation that instead LLMs are able are better able to deal with implicit knowledge rather than explicit one.

Um can I take this? Yeah. Yeah. Sure. I I I would say something slightly controversial for this crowd at least that I think the symbolic is red herring. It's not the question is you could essentially have a neurally compiled algorithm that is exactly equivalent to a symbolic algorithm and that would still be system two. Now and and in fact the symbolic part becomes important only in two ways. We wind up writing anything we do except for our manipulation and gestural knowledge. most of our uh declarative knowledge is written as either procedures or statements on in in natural language and so you need the LLMs need to pre-train models need to deal with them and then also if they're talking back to us they need to have a symbolic interface beyond that the question is not whether it is symbolic versus neural I think it's a red herring it's really a question of algorithms um with soundness and

completeness properties whether or not you're able to support that kind of a reasoning uh and is that going to just emerge by essentially you know in our particular current current case uh if I train myself on the unrollings of the same algorithm multiple you know multiple times uh the letter at least seems to be a very inefficient way to do it but it is a good thing for current LLM LRM technology and one has to remember that the currently this technology has become so powerful that they have they have won the software lottery. every company is actually trying to improve LLMs and LRM and rather than even other transformer architecture such as universal transformers which don't have these kinds of length effects built in but mostly we have the thing that we that already is working and we're going to improve this when is that going to change is anybody's guess but for now I would say we should at least understand

what we are what they're good for you know given the way we are actually doing these things as is saying touch is not full reasoning.

Yeah. So yeah. So we I think that we analyze what we have right now. But let's think a little bit more about the future. No. And we just have few minutes to do that. So we said that okay uh you have this different techniques. Some of them are providing sound reasoning. Some others are providing plausible reasoning. So what's the future here? Is just to use RL to improve LLMs to approximate better and better sound reasoning? Uh or do we need something completely different and the integration of these new what people call neurosymbolic things or plausible and sound reasoning? Olga, do you want to start?

Yeah. Um I personally think that there's no reason to think that we sort of scale our way out of this, right? So I think if we just continue doing what we're doing and and scale that up, we will not get anything that provides us with the kind of guarantees that in certain applications are absolutely crucial. I think we need something different there. Um which isn't to say that scaling things up and you know approximating correct reasoning better might not be useful um as well, right? But I think it's very important to keep the distinction, right? In fact, I would argue that an an algorithm or its implementation, right? A piece of code that's just a tiny bit incorrect could be way more dangerous in in practically than one that is grossly incorrect. And why is that? Well, because one, it's obvious to everyone that it's incorrect and we won't use it in in certain situations, right? Whereas the other

one, it might fail us when we least expect it and and that indeed could be very dangerous, right? And so therefore I think we're not going to scale our way out of that. Um at least I don't see why one should believe that and and as a corollery of that I believe it's very important to be a aware that that sound formal reasoning techniques exist and they've also gotten a lot better over the past decade. So progress in AI has not just been in uh in neural network based AI or or machine learning. In fact, there has been substantial progress also in these sort of you know more rigorous forms of reasoning that actually have been driving technological pro progress for example uh in in computing hardware. So it's important to know that these techniques exist. I think it's important to know at least for a good population of experts what they are. We need to continue teaching them. And then we need people who think

about how to how and when to use those and maybe how to combine them with other forms of of reasoning and and computation in order to perhaps get the best of both worlds in situations where this is is required. pointing always keeping in mind that there are still many and there will always be many many circumstances where very narrow capabilities that can be trusted are actually preferable over broad capabilities that cannot be trusted

okay do you want to comment on that yeah I think first of all I'm I'm I'm not I don't know why but I'm not very um interested in making prognostications about future I chose research just so I can invent future rather than just you know kind of make random prognostications. But the point I do believe is I think much of what is happening here is that there is this I guess the elephant in the room that we thankfully didn't talk about until now is AGI or super intelligence or as I would call ASDI artificial super duper intelligence which is my new company. But the question is what do the that all of that hype is based on this idea that there will be like one sizefits all solution that would do pretty much everything and that I don't see it anywhere on the horizon and it will improve there is there is this particular um substrate that we are using that might have aspects of other kinds of solvers

verifiers kind of compiled into the generator But there is no single one-sizefits-all. And another thing that we have to remember is this entire technology is highly highly dependent on what we provide. So it's like I was saying that uh yesterday that reaymon hypothesis will be solved by AI as soon as we find enough proofs of Raymon hypothesis and train the LLMs with that. And you know I was only being half facicious because in general this is a memory effect and it's a useful memory effect because what the humanity knows and what an individual human knows are like so different that the LLMs can bring humanity's knowledge to individual humans and in that sense they're amazing productivity enhancing tools but assuming that somehow they can solve everything that they will do science etc. New discoveries to some extent new discoveries are outside of the knowledge closure of the humanity to

some extent and and so and in fact none of these systems do action in the world without acting in the world you don't get new knowledge and this is something that's going to be holding them unless we find new knowledge and write it on the web so that they can train themselves on it it's not going to be enough but what they are they're extremely useful and we should use them for what they are good for.

Thank you. Okay. So um so I think we touched upon and we all dimensions here and also we took some of the questions and the topics that were mentioned in the chat. So I'll give you just uh the opportunity to say some final thoughts that you want the audience to remember and to take away from this panel discussion and to in general the discussion about AI reasoning. Olar, do you want to start? Yeah, I think there is a good reason why the chapter on AI reasoning is actually the first in our report. It's not because it's the most important thing there is to say about AI, but it is because it is a pretty fundamental thing, right? And it's a pretty fundamental thing that currently the the main thrust of progress is is somehow concerned with but hasn't led to, you know, a really good state of affairs.

has in fact led to a state of affairs that's a bit precarious because people generally tend to overestimate what um LLM based and and large reasoning model based technology can do in that respect. Right? So let me be very clear from where I'm sitting there is three important things. The first is excellent AI reasoning techniques of the kind that provide guarantees exist and are ready to be used and are being used to great commercial success. It's just less visible than you know what is mostly talked about today and that's very important. Secondly, they exist and are successful for a very important reason and that's that's because there are plenty of context where this kind of system is actually needed. Those kinds of capabilities are very much needed and science and engineering in particular are full of these sort of contexts, right? where it matters that you can correctly multiply numbers of arbitrary

length and precision where it matters that your sorting algorithm you know doesn't just provide something that looks approximately correct but that is 100% correct right um and the third point is emphasizing that there are weaknesses in the current technology isn't the same thing as saying the current technology isn't useful of course it's useful right but we there is actually no reason to say we can only have one kind of technology or the other. We actually have both. And what we need to get far better in in my opinion is to know currently when to use what kind of AI technology. And that requires knowledge of what there is and what its strengths and weaknesses are. And there we can get better and we should get better.

Thanks. All good. So Ralph, yeah, I I have a very short way of saying this that think of alchemy. Those people who were alive during the days of alchemy and thought alchemy is the answer basically failed. Also those people who thought alchemy means chemistry is useless also failed. Alchemy is problematic not because chemistry is a bad area but because some boss thought chemistry when prompted the right way will become nuclear physics. And that is the distinction I want us to remember. LLMs are great technology. LRM are great technology. Uh don't believe that they can do what they are not necessarily designed to do. Ask yourself what should be the limitations and you know work around them. Uh you want chemistry as well as physics. Um but you don't want alchemy guiding your research. Thank you.

Thanks Ralph. So let me tag the panelists. I think that this was very insightful and uh even innovative discussion around AI reasoning. I hope it shed some light or given some food for thought to the audience. Uh thanks also to everybody for joining us. Uh this panel has been recorded. The recording will be available very soon on the triple website. uh for the the same website where you find the report for the on the future of AI research where you can already find the previous two panels um and also will be available on the triple AI YouTube channel. So thank you all and I'll see you soon for another interesting discussion. Thank you.

Questions about this episode

Who speaks in "AAAI Presidential Panel on AI Reasoning"?

Franchesc Rosi (Host), Subaro Raalo (Panelist (referred to as 'Rao/Ralph')), Alexander Bonholber (Panelist (referenced as 'Olar/Olgar')).

What is discussed in "AAAI Presidential Panel on AI Reasoning"?

In order: 0:00 Triple AI, AI reasoning, Large language model, Subaro Raalo; 10:00 AI reasoning, Large language model, SAT, SMT; 20:00 AI reasoning; 40:00 Large language model, Triple AI; 1:00:00 AI reasoning, Triple AI, AGI.

What is mentioned in this AAAI episode?

13 people, works and ideas are mentioned, including AI reasoning, Franchesc Rosi, Large language model, Subaro Raalo, Triple AI, Alexander Bonholber. Each one on this page links to its own page with every episode that mentions it.

How long is "AAAI Presidential Panel on AI Reasoning"?

The episode runs 1 h 8 min; the transcript is about 11,362 words in 88 timestamped sections.

More from AAAI

About this transcript: the episode's spoken words as captioned on YouTube, split into timestamped sections. Reclip.Me locates every person, work and idea in the text, keeps the time it is first named, and quotes the words around it verbatim. Last updated January 24, 2026.