Transcript

The Illusion of Inclusion How LLMs Misrepresent African Languages and Cultural Contexts

From Microsoft Research · Episode page · Watch on YouTube

· 1 h 2 min · 9,905 words · 80 timestamped sections

This is the full, timestamped transcript of "The Illusion of Inclusion How LLMs Misrepresent African Languages and Cultural Contexts", an episode of Microsoft Research published November 16, 2025 (1 h 2 min, about 9,905 words). It mentions 14 people, works and ideas, including African NLP, Dr. Shams Sidin, NLP, Flores, Google Deep Mind.

What this episode is about

The discussion focuses on the limitations of large language models (LLMs) in accurately representing African languages and cultural contexts, specifically within Kenya. Dr. Shams Sidin highlights two decades of efforts in natural language processing (NLP) for low-resource African languages, emphasizing the need for equitable language technologies that truly reflect the richness and nuances of these cultures.

Key takeaways

  • LLMs are like bad interpreters at a multilingual conference—misrepresenting cultures while basking in the spotlight.
  • Over 2,000 African languages exist, yet LLMs barely scratch the surface of their richness and complexity.
  • Facebook's mishap: 'We got a baby' turned to 'We got a prostitute'—context is everything, and LLMs miss it.
  • As LLMs dominate NLP discussions, they're also sidelining the nuanced needs of low-resource languages. Surprise!
  • For Africa's linguistic landscape, the illusion of inclusion means we’re still waiting for true representation.

Who speaks in this episode?

What is discussed, and when

  1. – 10:00: African NLP, Dr. Shams Sidin, NLP, Google Deep Mind
  2. – 20:00: African NLP, Masakani, Google Deep Mind, ACL
  3. – 30:00: Flores, Common Crawl, African NLP
  4. – 40:00: Brighter
  5. – 50:00: Brighter, African NLP
  6. – 1:00:00: African NLP
  7. – 1:01:57: Masakani

Notable quotes from "The Illusion of Inclusion How LLMs Misrepresent African Languages and Cultural Contexts"

  • Dr. Shams Sudin is a Google deep mind academic fellow and advanced research fellow at Imperial College London and assistant professor at Bero University Cano. He is also a visiting professor at.
    about Dr. Shams Sidin · at
  • I moved to University of Pur and currently at serving as a Google did my academic fellow and advanced research fellow at Imperial. so this spans various you know from Nigeria.
    about Imperial College London · at
  • Rather LM's LLM because everything is now going to LLM in fact this year at ACL 60% of the papers that are submitted 67% they mention LLM at the title or the abstract. This mean that the attention is now moving to LLM, LLM. So, PhD.
    about ACL · at
  • What is the geographic spread and who are also the key contributors? the first one is the way we did this you know data collection is we collected 20 years of African LP papers…
    about African NLP · at
  • Sections and we manually annotated those papers. For example, this is one of my paper. Then we can see we have Google deep mine. Then we can count as one.
    about Google Deep Mind · at
  • Zalan university simbush but we can see masakani is also one of the which we already said because when we write papers at Masakani we put Masakani as one of our you know…
    about Masakani · at
  • First thing I want to show you here I always say that is that this is an example of common coral data set. you can see the percentage of English in common coral is 45% right…
    about Common Crawl · at
  • Languages but in four languages we audited which is house andu we found translation errors inconsistency and often machine generated text. Yeah. So for example for houseer we found that most of the text has been generated by Google.
    about Flores · at

Who and what is mentioned in this episode?

People

  • Dr. Shams Sidin — Person · Main guest / main speaker · 2 mentions · first at
    He delivers the seminar talk on 'the illusion of inclusion'—how large language models misinterpret African languages and cultural context—drawing on his research, datasets, and audits of training/evaluation corpora.

Companies & orgs

  • Google Deep Mind — Company · Affiliation of main speaker · 3 mentions · first at
    Dr. Shams Sidin is a Google DeepMind academic fellow, highlighting the company's involvement in African NLP research.
  • Masakani — Company · Community-led initiative · 3 mentions · first at
    A community-driven initiative that has significantly contributed to the progress of African NLP.
  • Imperial College London — Company · Affiliation of main speaker · 1 mention · first at
    Dr. Shams Sidin is an advanced research fellow at Imperial College London.
  • Bero University Cano — Company · Affiliation of main speaker · 1 mention · first at
    Dr. Shams Sidin is an assistant professor at Bero University Cano.
    I moved to University of Pur and currently at serving as a Google did my academic fellow and advanced research fellow at Imperial. so this spans various you know from Nigeria to Portugal to UK and back to you know.
  • University of Manchester — Company · Affiliation of main speaker · 1 mention · first at
    Dr. Shams Sidin holds an MSE from the University of Manchester.
    Timeline. as you can see I started from Nigeria. I did my bachelor in computer science. I moved to Manchester.
  • Houseera NLP — Company · Founded by main speaker · 2 mentions · first at
    Dr. Shams Sidin is the founder of Houseera NLP in Nigeria.
    And NERIPS and has received best paper awards right and served as area chair for major NLP conferences. He's also the founder of Houseera NLP in Nigeria and AAM data science…
  • AAM Data Science Academy — Company · Founded by main speaker · 2 mentions · first at
    Dr. Shams Sidin is the founder of AAM Data Science Academy.
    And NERIPS and has received best paper awards right and served as area chair for major NLP conferences. He's also the founder of Houseera NLP in Nigeria and AAM data science…

Products & tech

  • Flores — Product · Case study · 4 mentions · first at
    An evaluation dataset for machine translation that has been found to have quality issues for African languages.
  • Common Crawl — Product · Case study · 3 mentions · first at
    A dataset used for training language models, but found to have very little representation of African languages.
  • Brighter — Product · Referenced work · 4 mentions · first at
    A dataset created by Dr. Shams Sidin for evaluating emotion detection in African languages.
    Right. So the first one here you can see we have some of the work I selected here brighter which is evolution data set for many languages but we focus also on African languages and this paper won the best.

Concepts & topics

  • African NLP — Concept · Core topic · 8 mentions · first at
    Focus of the talk, discussing the progress and challenges of NLP for African languages.
  • NLP — Concept · Core topic · 2 mentions · first at
    Central theme of the episode, discussing the challenges and progress in NLP for African languages.
    Cano. He is also a visiting professor at Cameroon and also South Africa and York St. John University. His research focuses on NLP for low resource African languages promoting…

Places & events

  • ACL — Event · Referenced event · 3 mentions · first at
    A major conference in NLP where many African NLP papers are submitted.

Full transcript with timestamps

All right. [clears throat] Okay. So, I think we can start. Um, hi everyone and welcome to our seminar for this month. Uh we are super excited to be joined by Dr. Shams Sidin. Um [clears throat] he is going to talk about the illusion of inclusion. Uh how large language models misinterpret Kenya um African languages and and cultural context. Um Dr. Shams Sudin is a Google deep mind academic fellow and advanced research fellow at Imperial College London and assistant professor at Bero University Cano. He is also a visiting professor at uh Cameroon and also South Africa and York St. John University. His research focuses on NLP for low resource African languages promoting equitable um language technologies. He has published at you know various conferences like SCL EMNLP NCL ICLR and NERIPS and has received best paper awards right and served as area chair for major uh NLP conferences. He's also

the founder of Houseera NLP uh in Nigeria and AAM data science academy. He holds a PhD uh from the University of Porto uh uh his MSE from University of Manchester and bachelor's degree from uh Bay University. So in this talk Dr. Muhammad Muhammad will reflect on like two decades of Africa NLP um Africa NLP and community-led efforts uh that have advanced the field of uh NLP for African languages. He will discuss how large language models like misinterpret uh these languages and also the cultural meanings. And I think at this point I will uh pass it to you Dr.

Shams Sudin to share your screen and proceed with the presentation. Thank you very much for agreeing to give us a talk today.

Um thank you very much Meliss and thanks everyone for joining and also thanks for inviting me to give a talk here today. Um um can you hear me all? Everything okay?

Yes and we can also see your screen. Perfect. So um happy to be here to give a talk on the illusion of um illusion of inclusion and how LLM represent African language. Um so as has already been introduced my name is Shams Sadin currently Google Diffan fellow and um advanced research fellow at Imperial and also uh work a faculty member at Vay University. Um so a bit about my career timeline. Uh as you can see I started from Nigeria. I did my bachelor in computer science. I moved to Manchester. I moved to University of Pur and currently at um serving as a Google did my academic fellow and um advanced research fellow at Imperial. Uh so this spans various um you know from Nigeria to Portugal to UK and back to you know UK at sometimes. Um then after all these um I'm also very interested in kind of mentorship and whatsoever. So I have been involved in you know mentorship for MS student at M South Africa and

Cameroon also co-founder for Ario data science. I was NLP where we do mentorship and capacity I really um you know happy to be doing this stuff to see we have you know people coming from let's bridge village background to receive support that they may not have. Um so um the content of this is basically three things. I'm not going to talk much but only these three things. Uh the first one is going to discuss about the journey for African LP and then also we going to discuss about you know the idea now we have LLMs that you know um you know infiltrating anywhere and then we have the illusion that yeah they you know represent also African language. So we will see how or what happened. Do they really uh you know represent African language or they misrepresent and then finally uh I will do uh some presentation in some of our selected works uh you know uh regarding to African LP. Um but you know uh

everyone knows this right now now in AI in research in glob of south and we can see that all these LLMs you know everyone knows and we are working with them in terms of text generation image generation whatsoever but however all these uh LLMs are actually Mong's African languages uh this is one of the uh you know uh reports from science magazine that reports some of our work that how LLMs imprison and I always when I'm giving a talk I always give an example. Um if you attended my talk, you will hear one of this example. Uh the example is basically one uh one of my friend one time you know uh went to Facebook and posted her something like we got a new baby in house it means means we got a new baby but Facebook automatically translated to we got prostitute. Uh so you can see this one you know it really changes the context you know and also it changes the emotion. we got a baby is positive and

then we got a prostitute you know which prostitute it's something bad so this is how um these language models how these LLMs actually misrepresent Africa and also languages and also the culture um but um right now everyone believe that um LLMs they are redefining NLM NLP research uh as you can see this is actually Google translate uh from 2022 to this to date uh the word NLP and also LLM when being searched this is a trend as you can see LLM is now getting surpass the even the NLP so maybe one day in future we now stop saying NLP but rather LM's LLM because everything is now going to LLM in fact this year um at ACL 60% of the papers that are submitted 67% they mention LLM at the title or the abstract. This mean that the attention is now moving to LLM, LLM. So, PhD student, MS student, um when you want to do research, you need to think uh are we doing really doing classical LNP now or do we doing you know the new shiny stuff

of LLM? So, this is the current trend when you are submitting to ACL. In fact, some people will say oh this paper you know the LLM can do this stuff easily. So, there's current shift in this way. But the problem is African languages, there are more than 2,000 languages and all these things that we can see LMS can do amazing things on these languages. That's not true for African languages. And now how can we drive progress in African languages? So there's kind of visibility gap and we need to fix that uh because we cannot fix what we do not measure.

Right? So this is the idea now that in this talk I will do a very quick journey to uh African NLP which is a two decade journey of African NLP. Uh this is one of our work recently uh the raise of African NLP contribution contributors and community impact. Um in this uh talk I will talk about a little bit uh this stuff uh publication growth in African LP and what kind of contribution do we have? what are the people contributing? What is the geographic spread and who are also the key contributors? Uh the first one is uh the way we did this um you know uh data collection is we collected 20 years uh of African LP papers and we have 36 papers and we use LLM of course we are in the era of LLM we use LLM for example when we crawl we used keyword for example uh from abstract and title we use Nigeria Kenya or we use language houseer if there is language houseer in the abstract or title all those so we have a lot of

filtering stuff where we collect all these papers and and down there we now comes to 2,4 2,400 papers and then we do some annotation. So basically this is a pipeline we did to collect the African NLP paper. The first one is the paper African NLP or not. Uh we use LLM to filter that one after collection. And the second one we use to classify what we call contribution which I will discuss in a few minutes. And lastly we uh c we use LLM also to categorize what kind of contribution all those papers are is machine translation ASR and a lot of contribution and I will discuss this a bit uh but the most surprising thing you can see that one of the key thing is that when you limit uh you know uh thing that you can do with language then you limit somebody's own world right uh but we can see now this claim is changing as we rightly put it the limit of language mean the limit on my wall. Previously in

2007 you can see uh Africa NLP papers as you can see they are barely visible right uh but this uh continued to increase increase and you can see a noticeable increase from this you know sluggish movement in 2019 and whatsoever and from here now as we can see African NLP reach contribution more than 2,000 contributed and over 400 publication only papers that relates to um African NLP as you can see um So if we go down into this graph to see what really happened that you know skyrocketed this participation for African LP we can see the very first year here we can see from uh 80 60 84 but it jumped to 152 in 2019 and also jumped in 2020 to 20240 but what happened this is the year where communitydriven initiatives comes into being such as Masakani and also our workshop that organized yearly Africa NLP. What this tell us? It tells us that in Africa the community drives the progress uh in LP and this also corroborate with um white

paper from you know um Microsoft and community that come together that the real progress in AI is dominated by community grassroot movement rather than by the universities as opposed to in you know other countries for example in UK you see grant in I you know the the this progress is driven by universities by you know a big cooperation but in Africa is just you know grassroot movement uh in this case we ask questions the first one we ask is that okay now we have seen this is driven by grassroots movement but one question we ask is that what are the top organizations supporting the development of African NLP now what we did basically is this we went to each and every paper one of those 1,900 papers we go to the acknowledgement sections and we manually annotated those papers. For example, this is one of my paper. Then we can see we have Google deep mine. Then we can count as one.

Luna, we can count as one. And we can see this is basically one of the contributors of you know different kind of entities that we are not saying this is the real picture. But what we are saying this give us a bul fact estimate that tells from the data. Of course we may miss some papers that do uh you know put some kind of um you know um cognition but also examples some African papers they may not put in the um you know they may put in fire firewall and also they may not be visibly uh the way we clone our data. So this may not be the real u you know clearcut um you know uh contribution from but this is what we have from our data set. We also ask another question that um we also another question that what are the top affiliated instit organization of the authors. So this is really really interesting in the sense that given this is a paper and we can see we have authors we went to the authors manually annotated all the 1,900

papers and now we rank them and interestingly you can see university of pritoria is basically one of the top but however we can see uh we have many university that are coming not from Africa right we can see kaj millions zalan university simbush uh but we can see masakani is also one of the which we already said because when we write papers at Masakani we put Masakani as one of our you know affiliation so this is kind of as well as I said uh it's not the real you know perfect you know uh analysis but this is what we get from the papers we have so we can see uh we typically uh collaboration in Africa actually uh expand the NLP uh study in African languages uh But as well um you can see we have different kind of research theme in an Africa. Uh you can see here we have efficient uh multilingual machine translation uh in 2025 you can see the diversity of the theme we have in African LP uh and this has keeping

growing and you can see in 2024 the diversity of the research has increased largely this is 2025 and I believe the diversity will increase and this is kind of um themes that we took from ACL where we identify for each paper we took we use LLM to given the um you know abstract given the headline uh the title of the paper we ask LLM to predict when when of these um you know theme which of these theme the paper belongs to. So this is basically how we do that. As you can see of course African LP is progressing um from few things now to you can see we're managing but now one of the question we also asked in this uh report is how can we automatically analyze contributions? Uh so what is contribution basically in uh you know in NLP. Uh so we created what we call Africa NLP contribution data set. Now let me just start explaining what is contribution. We all know contribution is achieving uh people made mention in

their paper. For example, you can see here uh one of the paper here our evaluation reveals a significant performance gap between high resource languages. So this is a contribution right. We can see from the paper they contribute something and this is uh this re a significant performance gap between high resource languages is what we call contribution statement. Yeah. So for each paper in the abstract we extracted what we call contribution statement. Uh let me go again. Uh as you can see here we have another contribution. We released two sessana LM translated benchmark. This is a contribution. What kind of contribution is this? Is what we call data set contribution. Uh what kind of contribution is this? Our evaluation reveals significant. This is what we call knowledge contribution.

Now we have different kind of contribution that I will show in the next slide. But we annotateed contribution statement from the abstract of all the 1,900 papers that we got and now we categorize them and try to find out what kind of contribution African NP people are doing. Are they always creating data set? Are they always creating method? Are they creating language model? Who knows? So we need a way to automatically classify contribution. That's why we have Africa NLP contribution data set. And you can see this is a pipeline that I already discussed. We do have um so the contribution we have two different kind of contribution. We have knowledge contribution and we have actified contribution. The knowledge contribution is advancing human understanding. So for example here we said our evaluation a significant gap performance between high resource. This is uh you know a contribution advancing human

understanding and then we have artifact contribution creating new reser system. So this is artifact contribution where we said um you know we released two new data set. So this is artifacts. So you can see we have two different kind of contribution knowledge and artifact and we categorize them. Uh you can see for knowledge we have knowledge, data set, language, method, people, tax these are different contribution we have and for artifact we have data set we have method we have tax. So for example for data set here you can see what kind of the contribution is this introduce a new NLP data set right um so these are different kind of contribution this an example also contribution that you can check in our paper um so why do we need um African NLP contribution data set now for we take all these data uh these papers 1,900 and we annotate each sentence in the contribution and we classify it as whether knowledge

contribution whether artifact and This will allow us to understand what progress has been made and rebuilds the gap. If we understand okay African NLP they are not creating new models African they only focus in creating data set then we say okay there is a gap. So we created this data set manually annotated and then we do uh kind of benchmarking contribution and the goal is to automatically classify contribution uh from defined five classes. So you can see these are the five classes. We have language method. This method is a new method uh created a contribution or new method. U this is uh can tax is a new task. For example, somebody create a new very new task. This is knowledge about tax. So we can see we have eight classes that we classify and we can see that here you can see Afro SLMR did a good job and also cyber did a good job. One of the key thing you can consider look at is that cyard look a good good a good

job in K method. K method means the methodology and cyard was trained on uh you know scientific papers. So that's why it has a good performance. You can see that also it has a good performance but for appro SLMR because it's African centric language model it was able to have a better performance for knowledge people because in the Astra we automatically mention maybe people from Africa from you know housea and something like that so it's really good uh also for data set obviously the data set we annotate they are all from African African country and you can see the performance is good so our African LP data set was benchmarked. We created it. We have trained set and this test set and evaluation set and this data set can be used now to further be able to uh classify what kind of contribution we have in NFP community. Um yeah.

All right. So um again one of the key thing here we have seen we have a lot of contribution here but uh there is a paradox of progress um in the era of NLP. uh we can see two decade uh NLP moved from very simple few papers and now to close to 2,000 papers but yet foundation models still under represent African languages uh only few supported um you know across models uh I think uh from one of our paper the state of large language model which won the best paper award at the blended end I think we itemized I think only 43 out of 2,000 languages that are included in all the current large language model So this mean that this progress is at the periphery. Um so exclusion at the core AI of global AI. Uh we can see everyone we said we have low resource languages but for African LP this progress is at the very we are still yet to go into the deeper you know models inclusion for all the model availability.

Now have a look at the progress of African LP and then let's move on to see uh the second part that I call the uh you know the illusion of inclusion. Now what that means uh I have what we call I call triple gav uh that actually uh limit African languages. The first one is pre-training data. Uh so the pre-training data is bias and noisy. Number two, evaluation data. They are inaccurate and unreliable. Number three, culture missing the cultural blind spot of LLM. Um, so let's dive in into one of them and see what that means. So the first one is pre-training data which are bias and noise. So we know that for every LLM we have one of the key things to have a very good data to have a very good u language model um you know is as good if you have free training data. uh but for African languages uh there is an issue most multilingual corporate that used to develop or to create language model um they do not uh contain African

languages in fact if they include African languages the data is noisy uh some of them are translated direct translation um some of them they may limit domain for example they are mostly you know Bible coming from a Bible data set and when you train your model on this Bible uh related data your model would be biased to you know domain that the model was trained. So in this case African languages become statistically invisible during LM pre-training. That means uh when you do stuff for African languages you may not be able to get uh on par performance with uh languages from um high resource languages. Now let's look at a very few examples some of our work that show this idea. The the first thing I want to show you here I always say that is that this is an example of common coral data set. Uh you can see the percentage of English in common coral is 45% right uh French uh 4.7 German 5.7 other European languages

at this and you can see where is African languages you can see down here we had other a Asian other Asian right and you can see all other right all other is what 0.05% 05% Arabic 0.6 and this is what we call um data common crawl that is a data that's been crawled from all the internet that is internet archive and the percentage of African languages that you can see this is the percentage um what is it 0.05% not all African languages but we are lump into what we call other meaning African languages are inside 0.05%.

And this is one of the biggest you know data set common crawl that is used. So you can see there's actually bias in pre-training data. African languages are not involved and definitely they will be missing. The performance will be poor. The performance will be uh not on par with high resource languages. But not only this let's look at one of these work u which we did uh since 2022 uh quality at glance. So in this work what we did we manually audit the quality of 205 languages specific corpora including CC align paraclore and all these ones we manually audit them uh to find out what is the issue with this data set and we find some systematic issues at least 15 kofa from Africa have no usable text inside them and significant uh less than 50% sentences are of uh of acceptable quality and many are misle what I mean by misle is that you can see a language is called housea but is level as swahili

and this with the data set will be taken and train a model with with housea data as right so you can see this is a really really an issue where whenever our data set is not is noisy is not of high quality instead to create a positive stuff but it end up creating negative impression that what I call representation washing. What is representation washing? um community. We Africa NLP people we feel the sense of progress. Housa, Euroba, Swahili is all these languages are included in common crow right but this common curl data set the data set there they are of quality right and you know when they are benchmarked they may exaggerate the model for performance making low resource NLP appear more so than it is.

So when the data is not quality, it's not good and the people are using the data set and now given evaluating model or evaluating whatever and they give a false metric which does not and this effect could result in product uh productive effort being redirected away from these languages. So if we now have all the world uh big data set they say okay we include all the African languages okay people would sit down okay African NLP problem is solved the data set is solved now let's move on which is not true because the African NLP data set are not good high quality enough to make sure that they uh you know work as expected.

Um another thing from our paper the self of large language model we found that um Africa has 23 writing systems. Yeah 23 writing system but only three are represented in uh current LLM tokenizers. Um so the Latin the Arabic and gay these are the only three uh you know uh represented in model tokenizer that we have today. And then in that case most African scripts are invisible to LLM. We said 23 that means 20 are absent in all the language models. Um um um so hello somebody is disturbing us. Okay. So most African scripts are invisible because as we can see here we have 23 writing system but only three are now available in you know model taken. uh missing script of course uh broken tokenization as we know and the loss of meaning and semantic noise during training so which is really really a problem uh that pre-training data uh for African languages is bias and noisy now having understand the one of the um

triple effect that I said uh which is pre-training data let's move on to the evaluation data which I said they are inaccurate and unreliable now if we have our training data pre-training data is not good. Let's assume the pre-training data is good. But often at times even the evaluation data is not correct. It's not good. Let's look at it. Um so most multilingual benchmark often will represent African languages. Why? because they are often you know machine translated and if these uh benchmark data set that I use are machine translated then the claim must represent model capability of African languages because they are translated benchmark.

one of our work uh collecting flores evaluation data set for four African languages. We did uh we basically uh we basically took um you know uh Flores uh data set which most of us know Flores is one of widely used evaluation data set for MT especially for African languages but in four languages we audited which is house andu we found translation errors inconsistency and often machine generated text. Yeah. So for example for houseer we found that uh most of the text has been generated by Google translate and this mean that all the evaluation that have been been done using flores data set has been misrepresenting African languages and of course if the evaluation set is broken the claims for model performance are all broken. So all the previous you know model 5 performance that has been reported using florex data set are broken. Why? Because those languages, those data set they are not of good

quality. They are basically uh you know there are a lot of inconsistency often some of them they are not real they are machine generated. So the model performance is actually uh broken which is something that I found really funny that um a lot of people from Flores I mean people are heavily using Flores data set to evaluate their uh machine translation model which uh we now know that um there has been a very uh serious issue with Flores data set and I will encourage many people uh if you are interested to take your data data for your own language Flores data contain widely many many African languages take your data from your language and try to audit that data and try to find out what happens with your data set and this is something we are working to extend our work to more African language and you can reach out if you want to uh you know extend uh or validate or audited the data for your language. Uh moving on. Um

this is what our recommendation is. Uh future translation airports uh particularly in Loros langa uh protect the active involvement of native speaker at every stage of the process to ensure linguistic accuracy and cultural level. The reason why I said this is the plur data sets have been you know created manually. It's not something you know that actually uh automate created automatically. know they hire people pay people to create that data set but those people that are hired may not be the native speakers therefore they end up translating of course um I was one time a manager for a project and I'm leading housea and the person that is um you know I came to the project that worked on house he's not a housea ground he's not from housea and what he was doing is translate the data from uh you know from other from English to housea using google translate and now he give the data to body you know to look at some

correction and edit that. So our own recommendation is that whenever we are creating data set for African languages the very first thing is we make sure that the people that are creating data are the native speaker people that speak the language people that know the culture people that could depend the language wherever whatever it's written about the language they know the culture and know also every linguistic uh you know issue that relates with the language. So this is about evaluation data. We have now seen the pre-training data is broken. Uh evalation data is broken. Then another thing about evaluation is that most African languages are focused in simplest classification. When we do evaluation for every for LLMs we can say okay for LLM um we have sentiment classification, headsp speed detection whatsoever or we can say we have name entity recognition.

All this one I can see they are classification problem. But for African languages we have very very few uh data set for generation reasoning QA. So this is also uh a missing benchmark that we it's not possible to compare the African language with other high resource languages because we don't have shared benchmark to show the progress. So this is also one of the key issue that um um we need to close the gap to have you know more complex evaluation data set rather than the simplest classification uh benchmarks. Then the last one uh is uh missing culture. So LLMs can produce African text. Of course we know you can write it can produce African text to some extent but linguistic fluency is not cultural understanding. Yes. Um someone would say yeah um chip or any other can produce a quite fluent text in Swahili. Yeah. But does it really capture also the uh you know cultural understanding of the

people? No. Uh in many many cases no. This is one of our work called blend a benchmark for LMS um on everyday knowledge in diverse culture and languages. And uh if we have cultural errors, this can lead to hamper stereotype. So let's look at one of our work for blend. Uh so this work blend comprised of 52,000 question answer appears from 16 countries and we can see the performance we evaluate on English on from culture and US uh is 79% accurate and only 12% on America and house. So as you can see the culture even check GPT GPT now we have GPT5 and whatever but you can see the gap [clears throat] is really great uh it's it takes a lot of time to close this gap 79% and 12% accuracy on culture is really huge gap as you can see this is an example of the questions we have for example one of the question here we can see the first one here if you look at it here what is the most common eating food

in sport stadium while watching games in the US. So now what we did was that we localize this questions. So in Nigeria we can say what is the most and also in our own language in house. We write this prompt in house this question and answers in house and also reflect our own culture and we found that LLMs actually misrepresent or cannot capture African culture. The only thing we can see is that it can do 20% which is unacceptable rate if you have 79 to or 80% for uh you know uh English from USS better in English than the local languages. And another thing we found out is that uh all these questions we have we have them in two. We have the questions in English and then we have the questions in local African language.

And we found that even though the local language for example housea uh the uh the culture is written in house LLM can not be able to understand the culture. It understand the culture more when we translate these uh written multiple questions from house to English. So LM understand has better performance in English rather than on the local language. Uh this paper also blend won the best paper award uh uh one of the workshop at new rifts I clear. Um yeah so this is actually the triple gap pre-training data is vas and noisy evolation data they are inaccurate as we have seen and unreliable. One of our work that we audit flus data set we have seen a lot of inconsistency. uh some of the data are machine generated and also culture missions um cultural blind spot of LM we have seen that LLM cannot be able to uh answer some specific questions uh from African languages only 12%. And what do can we do? Uh, of

course we need to put huge effort for pre-training data reliable event uh evaluation benchmark and country grant and um understanding and thank god um recently we have a lot of grants coming from to Africa thanks to Bill and Melinda and um foundation they have put some effort to see uh Africa NLP is growing and there a lot of traction toward that creating more more data set for African languages and these are some of the you know things that I have recently discussed about uh the triple gap uh so what is missing in preion clean and diverse data um and if we don't have we can have bias and for evaluation we need reliable benchmarks so we need benchmark that is reliable if not we will have distorted performance you will report that uh the accuracy is 79 or 80 for house but initially if we look at the data set you are repres misrepresenting the uh performance and lastly the culture We need to have data set that um we need

to have data set LLMs that are culturally grounded. They can understand culture from African context. And then finally um I will present some of our work that we have been doing in Africa. Um selected works. Right. So the uh the first one here you can see we have some of the work I selected here brighter which is um um evolution data set for uh many many languages but we focus also on African languages and this paper won the best paper award in ACL 2025 um source paper um we have Afro SLMR and word translation in the era of large language model I will talk a bit about these and maybe skip some of them So let's go into the brighter. So this is a brighter uh which basically uh is um we try to see is it good LLM can detect emotion in languages uh from Africa and beyond.

uh uh one interesting thing is that this uh project uh the data set we created was also used for semiable share tax uh collocated with ACL. Uh interestingly the both the paper and also the share tax god award are the best tax paper and also base um uh paper award and if you are interested um you can check the project here but let's dive in into the what emotion human communication is definitely emotional right um uh people will share some happiness anger and whatsoever but multilingual and cultural challenges what you perceive as happiness or sadness or fear in one culture may be different in another culture. So it is very difficult to uh you know uh universalize the emotion from different culture. Uh what we did in this project we said okay we are going to focus on what we uh term perceived emotion.

What emotion? Most people some people will say okay this is not the emotion but what we agree is this what most people will think the speaker may be feeling given a sentence or short snippet what most people will perceive the emotion if most 80 60% this is happiness then we come this sentence or this um uh sentence is perceive as happiness. So this is what we define our tax and we created said more than 100,000 instances we created and predominantly 12 of these comes from Africa uh from 28 languages and you can see here we have many many of these languages as you can see them and um uh the data set was created from uh different uh domain uh speech social media literature and reviews and this is the brighter data set distribution ution. We can see we have anger, disgust, fear and you can see we have uh the distribution is not equal in all the classes which of course something that we um which is something that is true.

Um yeah so we annoted more than 100 instances from this data set as you can see them uh the classes are more than 100,000 and the data size vary from 1,600 to 7,600. So some of the classes for some languages as you can see these are the languages here some the sizes vary from one language to another and also variation in distribution. Uh we have also variation in distribution of the data set. Uh some of them you can see some languages may have anger which with a lot of uh you know number of anger some of them. So there is variation toward that as well.

Um what we found out is that number one is that overall model to perform better when prompted in English. What that means is that when we have a prompt in house we model we use that and we use that to ask emotion for that sentence. The model perform low than when we translate that sentence into English. That is it. Overall modest perform better in English and also you can see here um we have the uh performance you can see everything inside is low resource all African languages you can see all African language but you can see here outside we have Hindi for example and you can see all the um you know uh high resource language having better performance and low resource with low performance and finally for brighter we concluded that LLM still struggle with predicting passive emotion and um you know they have lower performance in low resource African languages.

Another work we have is uh recently is um what translation in the era of large language model. We know now machine translation um you know elements show a good performance for empty at sentence and document level right you can put uh you know for zero shot you just put it for zero shot you can have a very good translation but word translation is also crucial especially for low resource NLP because um word translation you can create bilingual dictionary you can create many other things so we what we did we evaluated LLMs for what translation and we average 106 languages including many many African low resource languages and we benchmark against plex. So for any language that we use LLM to translate uh given uh we give about 40,000 words in English and we ask the LMS to translate to housea and then we have pal which is a dictionary multilingual dictionary with more than 2,000 or 5,000 languages I think. Um so

we benchmark for any prediction from LLM. We compare with content from Plex and see what is the performance. We also compare the performance of LLM with uh Google translate traditional machine translation system and we found a very um huge card performance. So performance comparison we can see here that this is high resource language and this is low resource language. So for gamer Gemini we can see it's really pretty good. We can see it is only gamer 2.0 that has a very good performance up to 36% to try to translate African languages uh you know words into word that map with you know Google uh the uh gold standard from plex and we can see a very uh poor performance in all other open source you can see that but only two things we can see Gemini and Google they have a very good performance right maybe they they share some training data of course maybe uh yeah so we can see here that close

source models here from Google translate and Geminina they have better performance compared to open source such as NLB. Um here also we have peer wise agreement and here what we did is we tried to find out which models give consistent output. So we can see here for Google and Gemini their output always agree up to 23%. So this is also a uh a very few argument uh we have and the last but not least um one thing that we trying to see is that earl we have large language model large language model but finetuning still wins in African languages through what we call continual pre-training of African languages so we need to have maybe smaller we need to rethink we don't need to go far to say we need to have a very huge LLMs and whatsoever let's start from the beginning. So one of our work Afro SLMR which has been accepted at ENLP is that we did what is called domain adaptive pre-training and tax

adaptive pre-training. That means we take a an existing pre-trained language models and we continue training on social media data for African languages. So we create a corpus called Afriocial Corpus which is a large scale social media data for African languages and we did continued pre-training. The first one we did domain adaptive pre-train. We adapt domain and the finally and then we do the second one we did adaptive pre-train and we find out that uh you know the performance for apprehocial SML are consistently outperform GPT4 Gemini and Lama. Uh that means continual pre remains the vital resource for culturally African languages rather than GPT4 and um other huge LLM. But what is the intuition here? We can see GPTO very huge very large but Afro SLMR social is very small and it outperform all these stuff. Uh this is also a work from uh Ojo and from debit as you can see they also show that appro SLMR actually works

uh better if we fine-tune them. Uh it outperforms LLM such as GP2O Gemini Llama and whatever. So this is interesting because everywhere we are talking about LLM LLM GPT but this GPT as we can see um you know smaller models are much better than them. Now what is the direction? Can we still you know continue fine-tuning smaller models until the large language models you know have a good performance what is the way forward so this is something we can see that for me I said progress for African LP um must begin with linguistic infrastructure not parameter inflation we don't need to go and say okay huge billion parameters but before we scale up model we must scale off languages that's the point before we must scale up any parameters We need to scale up the languages. We need to scale up the number of data set. We need to scale up the number of languages we uh you know uh ours can understand rather than

parameter inflation. So this is one key point. And finally um what we have seen here we have seen that African NLP um collaborations such as Makani have transformed the African NLP land landscape and therefore grassroot community are the true infrastructure of African NLP and therefore we need more community-led effort to ensure languages are not left behind and that with this I conclude with saying inclusion is not automatic. We must design for it. We must come together. Uh everyone uh the scale holders uh the linguist the machine learning engineer the NLP everyone we must comes together. Um the big text comes together uh to support the African languages. Uh with this I think um uh uh I will say thank you very much. Um coming to the my last slide and um I'll be happy to take some questions.

Thank you very much and that was wonderful presentation. Uh thank you very very much. Uh Shamsin we have uh questions and yeah please we also have like time uh for questions. Oh um Ner you have your hand raised. Do you want to go first and then u you can just come off mute and uh ask your question. Yan you will be next.

All right. Thank thank you doctor for this wonderful presentation. Thank you. Please doctor, what are the potential consequences of the LM's limitation in African languages and cultural context for user particularly in the area like education, healthcare or governance? What are the potential consequences? Thank you.

Right. So um he's asking what are the potential consequences for airlines in many subfield if they are not performing really well. Um so what I said here is that um if we have um excuse me uh I want to show you. Yeah. So if we have these cultural errors lead to harmful stereotypes. Now if we have LLMs that actually does not reflect African culture, it may lead to stereotype, different kind of stereotype. Now maybe we can say okay what kind of foods um people from um Kenya eats um or what kind of you know uh you know what kind of uh uh religion do they believe in? Maybe you can say they are all atheists, right? Um, of course we know we have uh, you know, the we have Christian, we have, you know, Muslim whatsoever. But in an essence, what I'm saying is that cultural errors that LM does not reflect, you know, capture African language can lead to many many stereotypes. Yeah. Uh so for example we did one of the current work

we are doing um is that we are trying to look at what kind of um you know stereotype it can be uh you know a stereotype from you know women uh given a text you can ask LLMs to do some stuff but it can always report back to something regarding to men. So this is a stereotype actually about uh you know that gender stereotype uh stuff. So we have different kind of stereotype and this is one of the key thing.

Okay. Yan do you want to go or I I can just read for you the question if you want.

No super happy to go. Thanks Millisent and uh and good morning or afternoon and thank you very much Muhammad for the presentation. Um my question was that at least the first one on the do you have any correlation on the type of funders and funding that we have and the the type of research meaning do you see any interesting patterns between uh who fund for what uh and notably I'm particularly interested you know in this stereotypes topic in the sense do you see local institution researchers and maybe companies interested in that whereas my feeling or perception is that it's mostly global organizations that are mostly interested in the bias reduction and stereotypes but it's not based on data it's purely intuition so yeah I would like to see if you have any data

um yeah are you talking about uh this one not actually the contributors but mostly the founders I think you had a slide yet these ones and the the correlation with the type of research or actifact or knowledge that has been produced.

Ah, okay. So, I don't know. It's probably not one of the slides you have, but I'm wondering if you did like some correlation and you found like interesting patterns coming out.

Um, can you come over the question please? Maybe I'm missing it. Sorry. So, so you see you have different founders who who provide funding for the type of Yeah. research and then after you have a different type of artifacts or knowledge that is produced

but I'm wondering if you see any interesting correlations between where the money is coming from and the type of research uh that is produced

or the type of knowledge or artifact that is produced if you see any correlation that is interesting. Um so well actually I understand the question now but um we haven't explored that question to see if there is any correlation between the artifact and the funders what the people um try to that that's something interesting to look at to see what maybe funders are more interested in uh that is why they put more effort they fund stuff and that but what we found out is one that um for African LP uh generally the data set is what people do uh the artifact so data set um I think I slide or maybe in our paper for African NLP what we found out was that the artifact is basically the uh data set. So the data set is what we find at you know is what most people and also in the knowledge uh uh inclusion is what we found at uh you know African LP most people do.

Okay. Very clear. Thanks Mohamad. Yeah it will be great. Uh Dan uh you can go Dan Lami you can ask your question or maybe it's a mistake your hand is up otherwise Daniel

can you hear me? Can you hear me? Can you hear me? Yes. Yes. Uh thank you doctor for this very interesting uh presentation.

It was quite interesting and I think uh many people will believe with me here that uh we've going to take a lot of things back up. Um my question is this and I quite agree with you where you've mentioned about the infrastructural uh deficiencies especially linguistic infrastructure. Hello can you hear me?

Yes. Yes. Yeah. Yeah. So, so uh this is uh what we do here. For example, basically for me, I work on machine translation and uh uh you mention uh quite a number of fine-tuning strategies most especially the domain adaptation

and uh that is very quite interesting because uh no matter how many finetuning you go along with it uh we all agree that uh the low resource African languages are lack in data set completely. So we need to show all this uh data set and one of the best way to do is practically doing that finetuning the contining and then do adaptation finetuning that quite agreed. Now the question is you made mention of flourish data set being noisy being biased and is not even uh credible for evaluation. So what happened at this point for someone who is working for example in the tax of machine translation.

Mhm. Now if it is agreed that some of these benchmarking evaluation data set are not adequate enough to be able to evaluate some of those things. What is your proposal? What is your preposition? What do you think will be the next thing to do? Could you also involve in those uh curated data set you are using and then make them as a benchmark evaluation or could you look at other other crawl or other data set and then make them as a benchmark for your own tax?

Yeah. So um as I said um you know many people many organization different people uh sit down to create evolution data set but the problem is that people that are directly involved in these you know languages that know the culture are not involved. Um what happen now if they are not involved anybody can create that data set and the evaluation data set and the data set will not be of good quality. My recommendation is that whenever we want to create um you know use any evolution data set we must make sure that we check the quality of the data set. If the data set is not qualitative then we need to you know audit it and now you know do the evaluation on the data set. Of course uh many people will not stop creating the evaluation data set. Everyone will create but the question is how qualitative that evaluation data set is.

It is late for us our community African NP community you for example one of our work that we audit florex it is late for us to go and audit that data set to make sure that you know the data set is of high quality

thank you very much Muhamad and then um Gimal you're next and sorry guys we are like out of time but yes uh we'll stay a little bit uh longer if you have time. I think the questions are really interesting. Muhammad then Gimal.

Hi Shamsadin. It's wonderful to see you. It's been a while. Um Shamsadin, this has been a really interesting talk. Um I want to sort of come at it with a little from a slightly different perspective which is usefulness versus uh representativeness. [snorts] Um, even with GPT3 and the early GPT4s, they may not be very good at Swahili, but I can get them to do very interesting and useful things in Swahili. Um, and then domains where we lack resourcing become interesting. So um domains like education and healthcare um where because of the model sheer size and the amount of information that it pulls in they actually become quite useful um but they're not necessarily going to give you perfect um house or somali or raharik but they are useful so I'm interested in this boundary in your mind between being useful and being representative.

Mhm. Yeah, that's interesting. I I think um uh for all aspect and in all entirety I can see that um we cannot make a blanket statement that this they are not useful the LM at this point in time. Uh because in some sense uh you know from the explanation you did you said some stuff they can be useful to do some other stuff but um I think um one of the key thing that I discuss here is about the cultural blindness. um they may not actually uh carry our own culture. Uh so I'm looking at it from the perspective of culture not usefulness in other aspect. So if we look at for useful in other aspect yes of course they can do uh you know a lot of stuff they can be helpful for this LM and they have been helpful but if we talk about culture at some point they I think they have we have still long way uh to make this language model culture you know um uh culturally with our own culture from Africa which goes into the other

question that I had which is we are not going to suddenly get pocket fulls of cash to have people annotate them and

the exponential sort of there isn't likely to be an exponential growth in text and interaction at least text in African languages so how do we what strategies do we have how do we incentivize to make that growth

that's an interesting question I think one of the key thing is we need to get um one of the key thing always for data set is to uh get funding right incentivized and you're sure like you know when we started Masakani for example people come from the community they just work together without funing credit data science whatsoever but now um I think um things have changed even in our community masakani as well um you know uh that kind of spirit of people coming together is um you know fading away but still we do that but I think the main thing that you know uh drive progress is funding um we need to have funding uh that comes uh as I said in my talk uh we recently you know started getting funding for African IP which is really Um, good. And I think it will make a really change in the African EQ community in the next couple of years.

Uh yeah. And point taken. I will join the next meeting. Thank you. Um, all right. Thank you. So, I think I'm actually you have to catch your flight Shams right?

Or do we still have time? I'm my flight now. Oh my god. I'm catching my flight. He was at the airport. Oh my gosh. That's quite interesting. Ah, I forgot.

I need I need to tell if you want to talk. Yeah. So, yeah, we'll talk next time. Thanks.

All right. I'm sorry. My is waiting for me. Flying back to right now. Yeah. Okay.

Thanks so much. Thanks to them. It's wonderful. Okay. All right. Thank you. Okay. Bye. Bye.

Questions about this episode

Who speaks in "The Illusion of Inclusion How LLMs Misrepresent African Languages and Cultural Contexts"?

Dr. Shams Sidin (Guest), Meliss (Host).

What is "The Illusion of Inclusion How LLMs Misrepresent African Languages and Cultural Contexts" about?

The discussion focuses on the limitations of large language models (LLMs) in accurately representing African languages and cultural contexts, specifically within Kenya. Dr. Shams Sidin highlights two decades of efforts in natural language processing (NLP) for low-resource African languages, emphasizing the need for equitable language technologies that truly reflect the richness and nuances of…

What is mentioned in this Microsoft Research episode?

14 people, works and ideas are mentioned, including African NLP, Dr. Shams Sidin, NLP, Flores, Google Deep Mind, Masakani. Each one on this page links to its own page with every episode that mentions it.

How long is "The Illusion of Inclusion How LLMs Misrepresent African Languages and Cultural Contexts"?

The episode runs 1 h 2 min; the transcript is about 9,905 words in 80 timestamped sections.

More from Microsoft Research

About this transcript: the episode's spoken words as captioned on YouTube, split into timestamped sections. Reclip.Me locates every person, work and idea in the text, keeps the time it is first named, and quotes the words around it verbatim. Last updated January 24, 2026.