Language inclusion with LLMs

What this episode is about

The discussion focuses on enhancing language inclusion through large language models (LLMs) for underserved populations, exemplified by individuals like Charmila, who lacks formal schooling but uses technology effectively. Key initiatives include developing language technologies for 1,000 spoken languages, ensuring representative data for inclusivity, adopting a flexible model design for adaptability, and contextualizing AI to reflect cultural nuances in deployment.

Key takeaways

• Language models must embrace diversity; one-size-fits-all approaches leave millions unheard and unrepresented.
• Building adaptable AI is key; modular models can flexibly evolve with new languages and cultural nuances.
• Open-sourcing regional data empowers local voices, transforming language tech from exclusive to inclusive.
• Hard data reveals a cultural goldmine; multilingual users showcase language diversity that's often overlooked.
• Inclusivity in AI is more than a trend; it's an ethical imperative for tech that truly serves humanity.

Who speaks in this episode?

• Charmila (Guest) · 1 min speaking
• Jeff (Guest)
• Nia (Guest)
• Unnamed Speaker (Host) · 12 min speaking

What is discussed, and when

1. 0:00 – 3:00: van, Pragati Medan
2. 3:00 – 6:00: van, Indian Institute of Science
3. 6:00 – 9:00: Bindy, calm
4. 9:00 – 12:00: ACL 2022, seagull

Notable quotes from "Language inclusion with LLMs"

• Do is like you know play quick clip which shows how when we ask someone to speak like you know how to go to pragati Medan which is a location in Delhi in that up Bihar region how that's like you know getting changed so please.

  about Pragati Medan · at 2:54

• Building these models this kind of Representative data is captured so in the project vanani in collaboration with Indian Institute of science we are that's why taking a region…

  about van · at 3:46

• Incorporate this type of data into model building and do that in a flexible and modular manner so that's where the calm approach comes in which is doing composition for…

  about calm · at 6:07

• So now moving on to the third part of how we can incorporate recontextualize responsible AI so this is the effort called Bindy and this was inspired by some of the seminal work…

  about Bindy · at 8:47

• Probe and like you know mitigate them so with this goal in mind so we had in this ACL 2022 paper we laid out a research road map on how we can think about fairness in NLP by…

  about ACL 2022 · at 10:20

• Engaging with communities is also important to fix those residual gaps so we have a data set and effort called seagull where we were able to significantly expand The Stereotype…

  about seagull · at 11:22

Who and what is mentioned in this episode?

Companies & orgs

• Indian Institute of Science — Company · Collaborator · 1 mention · first at 3:51
  Collaborating with the van project to collect and analyze language data in India.
  Building these models this kind of Representative data is captured so in the project vanani in collaboration with Indian Institute of science we are that's why taking a region…

Concepts & topics

• van — Concept · Core topic · 3 mentions · first at 1:30
  Project focused on collecting representative data for language models in India, emphasizing regional variations.
• calm — Concept · Core topic · 1 mention · first at 6:11
  Approach to building language models in a flexible and modular manner to extend to new languages and domains.
• Bindy — Concept · Core topic · 1 mention · first at 8:52
  Effort focused on recontextualizing responsible AI for different geographies and cultures.
• seagull — Concept · Referenced project · 1 mention · first at 11:27
  Dataset and effort to expand stereotype evaluation datasets for AI models.

Places & events

• ACL 2022 — Event · Referenced event · 1 mention · first at 10:25
  Conference where a research roadmap on fairness in NLP was presented.
• Pragati Medan — Place · Setting · 1 mention · first at 2:59
  Location in Delhi used as an example in the van project to demonstrate regional language variations.

Full transcript with timestamps

0:09 so I'd like like you to meet uh charmila whom I happen to know for many years now so she is 45 years old has no formal schooling she speaks Sali and asmis uh has a smartphone and uses a bunch of apps on the phone now the question is like all the advancements that we all have been enjoying because we know a few languages like English and a handful of other languages so what is the relevance of all of like LMS and how we can make them helpful for someone like charmila and hundreds of millions of like her around the world so with this uh inspiration at Google we are working on this thousand languages mshot where the goal is to build language Technologies like machine translation recognition synthesis and so on for 1,000 spoken languages around the world like Jeff also mentioned and that's a vision he inspired us to work on so in this talk I want to give you a few sampling of uh Explorations we have been doing uh

1:07 in this area and this is by no means a comprehensive list so when we are trying to build these llms and uh make them useful for larger number of uh uh people and users uh inclusivity and doing that in a responsible way are the two Dimensions that we are thinking about and within inclusiveness uh having representative data uh that cover the demographics is really important so we have an effort called van I want to talk to you about that then when we are building these models rather than building a monolithic model can we think of building them in a more flexible and modular manner so that it's more extendable to new languages and new domains so that's an effort called calm and then finally uh when we are deploying building these models and deploying them in different geographies and cultures making sure that they are uh adapted to the Target geographies like you know cultural nuances uh and

2:05 values are super important and we have an effort called bindii which is looking at recontextualizing responsible Ai and I want to give you an update on that right so these three parts basically van uh uh C and Bindy so uh so van is looking at uh so India is a a huge uh multilingual country uh like you know there are like 100 plus languages which are uh uh spoken by 100,000 plus people about 60 plus languages which are spoken by million plus people and as you probably already know that a particular language kind of like you know how it gets spoken is a uh is dependent on what region we are talking about right so even if you take like say canara the way it's spoken like you know in south of Karnataka versus north of Karnataka there is a difference so what I want to do is uh like you know play quick uh clip uh which shows how when we ask someone to speak like you know how to go to pragati Medan which is a location in

3:02 Delhi in that up Bihar region how that's uh like you know getting changed so please listen so as you can see like you know the same sentence like you know when we are looking at that Hindi belt region like you know there is so much of variation so it's important that when we are building these models this kind of Representative data is uh captured so in the project vanani in collaboration with Indian Institute of science we are that's why taking a region anchored approach rather than a language anchored approach so what we do is we have uh we work at a district granularity in the whole country and when we Source uh locally relevant images and show it to people uh from these districts and ask them to describe these images in a language of their choice which is super important because we are not being prescriptive in terms of saying that like okay describe this image in the handful of languages that we care about

4:24 uh and uh and all of this data is being open- sourced so ultimately our aspiration is to go to all 773 districts in the country and then collect 150,000 Plus hours of speech and transcribe 10% of that so if you go to uh van. i. ac.in will'll be you'll be able to track the progress uh of the project and currently we are in phase one uh where we have covered 80 districts so we are going towards the end of the phase one and we are starting phase two which is another uh 80 districts and we have already open sourced uh or publicly released uh 10,000 plus our of speech data and this is all multimodal data so you have like the images the speech and the 10% of that data is being transcribed and what you see on the uh right hand side here is the map from different parts of the country in phase one where we have uh sample data from and the colors are representing different languages that people actually spoke uh like know that

5:22 they express themselves in and what's been really heartening to see here is that when you give people that opportunity to express in language of their choice the diversity of languages that they use is enormous you probably cannot see the fine print here but we have seen uh like you know people you speaking in sambalpuri marari chatis gari like you know those languages we would have never thought about like you know if you were being prescriptive and telling them like you know speak in a handful of language so I think that's really exciting and please uh uh track the progress on this site and then we also have a booth here uh you can uh uh uh listen to some of the samples that the data has been collected so far so that's about representative data now how do we kind of like you know build incorporate this type of data into model building and do that in a flexible and modular manner so that's where the calm

6:13 approach comes in which is um doing composition for language models and we are thinking of this as an llm augmented llm so right now if you have a language model and you want to extend it to cover a new domain or a language then you have to do expensive uh like you know uh continued pre-training or fine tuning for that language which is expensive and it's not very scalable right because how how can you uh enable lots of people to contribute towards development of these models so motivated by those problems we've been thinking of uh this scal model where we have a anchor model uh which is the model on the right hand side that we want to augment uh with knowledge from an augmenting model right which is the model on the left hand side and we have some additional cross attention parameters that we learn while keeping these base models Frozen so that way you don't have catastrophic forgetting on these base models but then

7:07 you are also able to adapt to say new domains uh to enable new capabilities so just to give you one example maybe you are interested in doing numeric reasoning over key value arithmetic so things like what is the value of X1 plus X8 times xn where the key value knowledge is there in your this augment ing model on the left hand side and then the numeric reasoning capabilities is there on the model on the right hand side so in order to do well on this task you need to compose like you know capabilities from both of these two models and that's what Cal enables so if you're interested you can go and read the paper but we since we are talking about language inclusion we also applied this framework uh in seeing how we can extend a a language model for a new language so or a set of languages so in that case what we did was we had the base model the anchor model but then we had a separate model the augmenting

8:04 model which had expertise on a bunch of new low resource languages and then when we combine both of them uh through C and then evaluated them on uh language benchmarks in particular case machine translation we see that this composed model is able to do well for many of these languages and especially the improvements are there for the low resource languages so this demonstrates one way of how we can build more linguistically inclusive uh language models but in a composable and modular manner rather than having to train like you know one monolithic model and uh suffer from all of the problems that was also discussed during the fireside chat so now moving on to the third part of how we can uh incorporate recontextualize responsible AI so this is the effort called Bindy and this was inspired by some of the seminal work that was done by Nia and her colleagues so Nia is also here with us today and

9:03 this kind of like you know makes this observation that while responsible AI is important lot of that work has happened from an Western lens and when you are trying to apply these kind of models and deploy them across like you know different geographies and cultures it's important that we adapt them to those Target geographies and cultures so uh this is also not like you know some um abstract problem uh so when you when we tried using some of the Shelf sentiment analyzer so given a piece of string you say like okay know whether this is positive or negative and then we generated a bunch of sentences with different identities right so in this particular case the identities are based on cast so the only thing that's changing is like you know whether it's like Brahmin obes or shria or like you know other cast identities but the rest of the string Remains the Same and we are asking these models to predict like

9:55 you know what's the sentiment of these strings so what we find is that based on like you know for certain types of identities in this particular case like you know there's a significant deviation of sentiment scores compared to like you know other identities right and there was really no reason for these kind of deviations to happen so this is to highlight that like you know these kind of biases are there in our uh like you know models and we need to be uh able to probe and uh like you know mitigate them so uh with this goal in mind so uh we had in this ACL 2022 paper we laid out a research road map on how we can think about uh fairness in NLP by taking India as an inspiration and there are three dimensions that we're looking at in societal context technological gaps and value alignment uh again if you're interested uh like know you can take a look at the paper now one uh last point I want to talk about is in terms of

10:49 evaluation in this domain uh focusing on stereotypes so some of the examples that I'm going to use on this slide might be offensive to some people so please uh uh please note take note of that but the point here is that when we are trying to build this kind of like evaluation data sets we really need a complimentary approach of how we can combine like you know the scale that comes from llms uh along with Community engagement right because the llms are going to have like a certain types of coverages which is great for like initial scaling but engaging with communities is also important uh to to fix those residual gaps so we have a data set and effort called seagull uh where we were able to significantly expand The Stereotype evaluation data sets that's there uh by 5x compared to all other previous uh data sets of this kind and uh now uh this is able to cover one 78 countries and uh six continents so this is using

11:48 llm scaling but then uh when we are looking at uh like know the residual like you know stereotypes we are able to see that when you engage with communities we able to get additional examples that were not covered by the llm so I think we need this kind of complimentary approach of having both llm based scale and then how we can uh work with communities to kind of like you know be uh build truly representative uh data sets and benchmarks so I'll stop here but this is an exciting area and uh we are hopeful that by uh collaborating all of us can really bring down a language barrier and make sure that the girl in rural jaran or like the boy in subsaharan Africa has the same access to information and opportunities like you and I here thank

12:49 you

Questions about this episode

Who speaks in "Language inclusion with LLMs"?

Charmila (Guest), Jeff (Guest), Nia (Guest), Unnamed Speaker (Host).

What is "Language inclusion with LLMs" about?

The discussion focuses on enhancing language inclusion through large language models (LLMs) for underserved populations, exemplified by individuals like Charmila, who lacks formal schooling but uses technology effectively. Key initiatives include developing language technologies for 1,000 spoken languages, ensuring representative data for inclusivity, adopting a flexible model design for…

What is mentioned in this Google Research episode?

7 people, works and ideas are mentioned, including van, calm, Bindy, Indian Institute of Science, ACL 2022, seagull. Each one on this page links to its own page with every episode that mentions it.

How long is "Language inclusion with LLMs"?

The episode runs 13 min; the transcript is about 2,242 words in 14 timestamped sections.

More from Google Research

• All analyzed episodes
• People mentioned on Google Research
• Topics mentioned on Google Research