How LLMs might help scale world class healthcare to everyone

What this episode is about

The discussion explores how advancements in large language models (LLMs) can democratize access to high-quality healthcare, highlighting the challenges faced by individuals in underserved regions who struggle to access medical services. It emphasizes the importance of developing specialized AI models trained on quality medical data to ensure they align with medical practices and values, paving the way for universal access to expert healthcare advice.

Key takeaways

• In India, walking miles for medical care shows accessibility gaps - AI could redefine doctor-patient dynamics.
• While AI excels in medical Q&A, the real battle lies in convincing skeptics to trust tech over touch.
• Met Pom's multimodal capabilities reveal a future where AI can diagnose and interpret beyond just text.
• Today's AI can outperform some doctors, but will patients trust a screen over a stethoscope?
• Innovative tech democratizes healthcare, yet ethical frameworks are the unsung heroes of responsible progress.

What is discussed, and when

1. 0:00 – 3:00: Met pom, Nature
2. 3:00 – 6:00: Amy, Google LLC, Met pom
3. 6:00 – 9:00: Google LLC, New England Journal of Medicine

Notable quotes from "How LLMs might help scale world class healthcare to everyone"

• Worldclass medical expertise is available at the fingertips of everyone everywhere and especially you know with llms like gbd4 and bod we have general purpose intelligent…

  about bod · at 1:04

• Models to medical school and so with our work on met pomm and met pomm 2 which was published in nature we showed that these specialized language models can attain expert level…

  about Nature · at 2:11

• Multimodal Endeavor and so we then proceeded to add multimodel capabilities to met pom and so the Met pom M model which is the multimodel version that model is not only capable…

  about Met pom · at 2:57

• Field of AI dating back several decades but I'm very pleased to say that with Amy or articulate medical intelligence Explorer this is our latest research AI system we made…

  about Amy · at 4:55

• Built-in automated feedback mechanisms and this approach was heavily inspired by several Google research and Google Deep Mind breakthroughs over the years such as alphago and…

  about Google LLC · at 6:45

• The acity study that I showed previously and so in addition to the acity study we also evaluated Amy as an assistive tool for clinicians in solving complex diagnostic…

  about New England Journal of Medicine · at 7:54

Who and what is mentioned in this episode?

Books & papers

• Nature — Book · Referenced work · 1 mention · first at 2:15
  Nature is the journal where the work on Met pom and Met pom 2 was published.
• New England Journal of Medicine — Book · Referenced work · 1 mention · first at 8:04
  New England Journal of Medicine is the source of complex diagnostic challenges used to evaluate Amy.

Companies & orgs

• Google LLC — Company · Business focus · 3 mentions · first at 3:54
  Google is the company behind the development of specialized AI models like Amy and Met pom.

Products & tech

• Amy — Product · Business focus · 2 mentions · first at 4:59
  Amy is a specialized AI system for medical diagnostic dialogue, outperforming physicians in controlled settings.
• Met pom — Product · Business focus · 3 mentions · first at 2:15
  Met pom is a specialized language model for medical question answering, showing expert-level performance.
• Met pom 2 — Product · Business focus · 4 mentions
  Met pom 2 is an advanced version of the Met pom model, showing improved performance in medical question answering.
  Things we had another pre-print in collaboration with some collaborators at Stanford and so here in this work a metp 2based hypothesis it led to the development of a novel byic…
• bod — Product · Referenced technology · 4 mentions
  bod is mentioned alongside gbd4 as a general-purpose intelligent substrate in the context of AI progress.
• gbd4 — Product · Referenced technology · 4 mentions
  gbd4 is mentioned as a general-purpose intelligent substrate in the context of AI progress.
  Worldclass medical expertise is available at the fingertips of everyone everywhere and especially you know with llms like gbd4 and bod we have general purpose intelligent…

Full transcript with timestamps

0:09 so like uh many of you I also grew up in India and in you know some of the parts of the country that I grew up in uh for many people going to see a doctor was often simply not an option so it often meant like walking 30 40 miles in extreme heat or you know giving up on a day's wages or going without food and so I kind of knew people who had not seen a doctor in their entire life times and so this like really bothered me and I've always wanted to do something about this so even back in my undergrad a few friends and I we were you know building this app called ask the doctor anytime anywhere and so the name is very very descriptive and it kind of tells you what you were trying to do but unfortunately the tech and AI needed for such a thing to work was just not ready back then however the Arc of progress in Ai and particularly in large language models over the last decade I think it you know allows us to dream allows us to

1:02 dream of that future again where worldclass medical expertise is available at the fingertips of everyone everywhere and especially you know with llms like uh gbd4 and bod we have general purpose intelligent substrates and so these models are you know like a very smart friend of yours so you can talk to them about you know physics and poetry and philosophy and maybe some of them if they have not been lobotomized Politics as well uh just joking um but you know if you have a medical concern or if you need medical advice do you go to the smart friend of yours I really hope not I hope you go to a doctor someone who's undergone you know rigorous training and is licensed to practice medicine and so similarly if we are to use AI models in you know Healthcare why should it be any different we need specialized models we need models that have been trained on high quality medical data have been aligned to the values and practice of

2:01 medicine and you know have undergone rigorous training and validation and so this is precisely what we've been doing in our team over the last couple of years you know sending these Foundation models to medical school and so with our work on met pomm and met pomm 2 which was published in nature uh we showed that these specialized language models can attain expert level performance uh in medical question answering benchmarks in controlled research settings and so I think the nice thing about these results where where that these models were not only good on multiple choice you know medical exam benchmarks but perhaps you know more impressively uh they were also doing well on nuanced and long Fong medical question answering and one of the key results in this paper was that they were actually surpassing expert board certified Physicians on several clinically relevant axes such as you know factuality uh clinical reasoning

2:53 and harm and bias um but at the same time you know medicine is an inherently multimodal Endeavor and so we then proceeded to add multimodel capabilities to met pom and so the Met pom M model which is the multimodel version that model is not only capable of language based interactions but it can also interpret medical images it can chat to you about them it can you know ground its responses in the context of a health record it can write Radiology reports competently and it can also interpret increasingly new modalities of healthcare data that are coming online such as genomic sequences and then more on the biomedical Discovery side of things we had another pre-print in collaboration with some collaborators at Stanford and so here in this work a metp 2based hypothesis it led to the development of a novel byic model for hearing loss and mice which in turn is now undergoing experimental verification

3:43 and so this is I think a fun demonstration of how these large language models might actually accelerate biomedical Discovery and scientific Endeavors and obviously all of this work has over the last year received a lot of attention um but we at Google believe in not just you know bold Innovation but also uh responsible Innovation and so we've been working with many of the biggest Healthcare organizations uh life sciences organizations as well as with academic research Partners to understand the use cases and the opportunities but also the failure modes and limitations of these models and using the feedback to improve and I'm happy to say that we made considerable progress and just last month we made the Met pom uh family of models generally available via Google cloud in the US as medm um so met is definitely a useful medical language model and there's a demo over there and so if you want to

4:33 play around with it please do that however it was not optimized for Diagnostic reasoning and dialogue and as I alluded to in the beginning of the talk AI systems capable of you know diagnostic dialogue could greatly enhance the accessibility affordability quality and consistency of care however building such systems has been a long-standing grand challenge for the field of AI dating back several decades but I'm very very pleased to say that with Amy or articulate medical intelligence Explorer uh this is our latest research AI system we made actually made significant progress towards this goal and so we conducted a double blind uh randomized study where we had validated patient actors interacting with Amy or both certified primary care physicians via a text chat interface and so these consultations were designed to mimic objective structured clinical examinations or osis which are widely used in medical school

5:29 training and is known to be like the bane of medical students worldwide I mean they hate it but uh again so in this study what he found was that Amy actually outperformed physicians in this controlled setting on multiple axes of diagnostic dialogue as evaluated not just by the patient actors themselves but also specialist Physicians and uh you know in particular Amy outperformed the pcps not only on diagnostic accuracy and treatment management but perhaps more profoundly it was rated to be better on axis pertaining to relationship fostering and rle building with the patient and empathy and so beyond the RCT resols themselves I'm actually quite excited by how we developed Amy and so there's the saying that there is no substitute to experience in medicine um however if you look at most human doctors they're actually likely to only see an order of you know tens of thousands of cases in their entire lifetimes and this number

6:19 is far less for you know medical trainees and early career Physicians however the advances in of AI means that we can actually train these systems in simulation on several orders of magnitude more cases and this potentially provides us a pathway toward its superhuman diagnostic performance with these AI systems and so in particular for Amy we designed a selfplay based multi-agent diagnostic dialogue learning environment with built-in automated feedback mechanisms and this approach was heavily inspired by several Google research and Google Deep Mind breakthroughs over the years such as alphago and so in this uh you know environment we had Amy interacting with a simulated patient super ized by a moderator and receiving feedback from a Critic all of these played by Amy and so this environment actually allowed Amy to experience patients from you know wide variety of disease and condition and

7:12 symptom presentations but also other important aspects such as medical history uh socio economic background and even things like personality and so you could have Amy interacting with a patient who's worried or someone who's anxious or someone who is you know just adversarial and so you could have all those interactions and experiences and I think the nice thing about this framework was that with you know every iteration of the selfplay loop that we did uh Amy became better not only as a doctor agent but also at like playing a patient and also in terms of giving feedback and so this kind of created this you know virtuous continuous self-improving cycle and I believe this simulated learning process was one of the key reasons for the impressive performance that we saw uh with Amy in the acity study that I showed previously and so in addition to the acity study we also evaluated Amy as an assistive tool for clinicians in solving

8:04 complex diagnostic challenges sourced from the New England Journal of Medicine and so here what we observed was that in addition to strong Standalone diagnostic performance um Amy also demonstrated significant potential to assist clinicians and so while these results are still early I think what they suggest is that perhaps in the near future uh the standard of care should no longer be a human doctor alone uh but rather a human doctor aided by specialized LM such as Amy and so I think all of this brings us to the question what is the norstar vision with all of this what is the future and so I think as I alluded to in the beginning of the talk the idea of world-class medical expertise in the pocket of billions of people worldwide is no longer science fiction and I think that is what we are shooting for uh but I think in the process of doing that and encoding the biomedical Universe with Advanced uh AI systems such as Gemini

8:56 this the model that we going to end up with is going to look more like a world world-class AI physician scientist and so this is a model that's not only going to democratize access to care to everyone but I think will also help us improve our understanding of biology help design better therapies ease the burden of diseases and really help scale personalized Healthcare to billions worldwide and of course when such a powerful technology comes to fruition you know there are a lot of broader questions that we as a society need to Grapple with so I'm not going to go into details of all of them and feel free to come over and chat with me uh but I'm actually very very optimistic that we can do that perhaps nowhere is ai's impact going to be more profound than in biom medicine I think the future is incredibly exciting and can't wait thank

9:58 you

Questions about this episode

What is "How LLMs might help scale world class healthcare to everyone" about?

The discussion explores how advancements in large language models (LLMs) can democratize access to high-quality healthcare, highlighting the challenges faced by individuals in underserved regions who struggle to access medical services. It emphasizes the importance of developing specialized AI models trained on quality medical data to ensure they align with medical practices and values, paving…

What is mentioned in this Google Research episode?

8 people, works and ideas are mentioned, including Amy, Google LLC, Met pom, Met pom 2, Nature, New England Journal of Medicine. Each one on this page links to its own page with every episode that mentions it.

How long is "How LLMs might help scale world class healthcare to everyone"?

The episode runs 10 min; the transcript is about 1,795 words in 12 timestamped sections.

More from Google Research

• All analyzed episodes
• People mentioned on Google Research
• Topics mentioned on Google Research