The Unequal Future of Artificial Intelligence
This summer I started working on a project translating children's stories and some short math lessons into Dari and Pashto, for Afghan families who had just moved to the U.S. The plan seemed pretty simple. Take a story like The Lion and the Mouse, run it through a translation tool, clean it up, and put it in a small online library that a kid or a parent could actually use.
The first part was easy. The second part, not so much. With Spanish or French, you can paste a paragraph into a translator and get something that's usually close enough. With Pashto, I kept coming back to the same question: how do I actually know this is right?
That got me thinking about something bigger. AI is turning into part of everyday life - people use it for school, for jobs, for talking to doctors, for filling out government forms. But if it works great in some languages and noticeably worse in others, then it's not really a tool for everyone. It's a tool built for some people, and everyone else gets a watered-down version of it.
Whose Words Is AI Learning From?
In a previous post, I talked about how AI picks up a language by finding patterns in a huge pile of text from the internet. So the obvious next question is: whose text is in that pile?
Mostly English. One of the biggest sources AI companies use is called Common Crawl, which is basically a giant public archive of the web. English makes up somewhere around 43 to 46 percent of it, while most other major languages only get between 5 and 10 percent (Stimson Center). For languages spoken by millions of people, like Tagalog, Punjabi, Kurdish, Lao, and Amharic, the share drops below 0.01 percent (Tessema et al., 2024).
What surprised me is that this doesn't line up with how many people actually speak each language. Most people in the world don't speak English as their first language, but it still dominates the internet. Imagine walking into a library where almost half the books are in one language, and then being told to become equally fluent in every language using only that library. That's pretty much the situation these models are in.
It also comes back to tokenization, the "chopping system" I mentioned before. A model built around English chops up other languages in clunky ways. One study found that the exact same text can take up to 15 times more tokens depending on what language it's in (Petrov et al., 2023).
More tokens means it costs more, runs slower, and the model can hold less of your text at once. It's kind of like paying for texts by the word, except your language gets charged extra for saying the same exact thing.
Why Dari and Pashto Get Left Out
Dari and Pashto are the two main languages of Afghanistan, and tens of millions of people speak them (Rest of World, 2023). That sounds like a lot, and it is. But being spoken by a lot of people isn't the same thing as being written down online by a lot of people.
Both are written right-to-left in a script based on Arabic letters, and Pashto has a few extra letters of its own. Spelling isn't always consistent, and there are regional dialects with different words for the same thing. A lot of everyday communication also happens out loud or on social media, instead of on the kind of clean, formal websites that end up in training data. On top of that, it makes sense that decades of war in Afghanistan would make it harder to build up the schools, publishing, and internet access that usually produce tons of written text.
Dari has an interesting twist too. It's a form of Persian, so a model can lean on the much bigger amount of Iranian Persian (Farsi) online. That's cross-lingual transfer again, the same idea as a Spanish speaker getting by in Portuguese. The catch is that the model can end up using Iranian words and phrasing that sound a little off to someone from Afghanistan, kind of like a British word showing up in an American textbook.
Translators who work with Afghan refugees have noticed the gaps in very specific places. Machine tools they tested struggled with the words for certain family members in Dari dialects, and with military ranks (Rest of World, 2023). Those sound like tiny details. They aren't, once you think about who needs them translated right.
When a Bad Translation Actually Matters
In my earlier post, the worst thing a bad translation could do was make a tourist accidentally call themselves attractive in German. Embarrassing, but harmless. For a lot of people, the stakes are way higher.
Take asylum cases. In 2020, a translator working with a Pashto-speaking refugee found that a machine translation of the woman's statement had swapped "I" for "we." That one pronoun made her written application look like it didn't match what she said in her interview, and her asylum claim was denied (Rest of World, 2023). In asylum cases, small inconsistencies can be treated as signs that someone isn't telling the truth, so one wrong word can literally change someone's life.
The problem only got more urgent after the Taliban took back power in 2021 and huge numbers of Afghans fled the country. Some groups that help refugees now refuse to use machine translation for these documents at all and rely only on human translators (Rest of World, 2023; PBS NewsHour, 2023).
It shows up in smaller ways too. Think about a parent who just moved to Virginia and gets a school orientation packet, a permission slip, or a report card. If the tools they have are weaker in Dari or Pashto, they end up understanding less about their own kid's school than the English-speaking parent sitting right next to them.
And as more people start getting answers from AI assistants instead of looking things up themselves, speakers of these languages basically end up with a lower-quality version of the internet.
Can AI Help Save Endangered Languages?
If Dari and Pashto count as "low-resource," imagine a language with only a few hundred speakers left. More than 3,000 languages, over 40 percent of all the languages in the world, are endangered (Ethnologue). A lot of them barely exist online at all, and some are mostly spoken, not written.
What's interesting is that the same technology leaving these languages behind might also help keep them around. A group called FLAIR, led by Indigenous technologists, is building speech recognition tools that can be made quickly for endangered Indigenous languages (Mila). A Muscogee (Creek) language project is using speech recognition to turn recordings of fluent speakers into a searchable library, so learners can look up a word and actually hear it used (MIT Solve).
Still, AI can help record and teach a language. It can't make kids grow up speaking it. A language survives when people use it at dinner and with their grandparents, not when it's sitting on a server. Communities also want to stay in control of their own recordings, which makes sense, since outsiders have taken and used their culture without asking plenty of times before.
Who's Working on It
The good news is that people are working on this. Meta built a model called NLLB-200 that translates between 200 languages, about three-quarters of them low-resource, and it did 44 percent better on average than older systems (NLLB Team, 2024). They also made it open source so other researchers can build on it.
Masakhane, which roughly means "we build together" in isiZulu, is a grassroots group of researchers working on AI for African languages, by Africans. Since 2019, it's grown to over a thousand people (Ars Electronica, 2023).
Common Crawl has also asked speakers of other languages to send in websites written in their language so its crawler can find them (Common Crawl, 2024). And the researchers who found the 15x token gap argue that future models should be built with tokenizers that are fair across languages from the start, not patched later (Petrov et al., 2023).
What I Think
After going down this rabbit hole, here's where I landed. The language gap in AI isn't some random glitch. It's more like a mirror. AI is good at the languages whose speakers had the money, the schools, and the internet access to put billions of words online, and it's weaker at the ones that didn't. So without anyone really meaning to, it can make a gap that already exists even bigger.
That doesn't mean we should stop using AI for Dari and Pashto. Honestly, for my own project, machine translation is what made it possible for one high school student to build a multilingual library in the first place. But I've started thinking of it as a first draft, not the final answer. For anything serious, like an asylum application or a medical form, a person who actually speaks the language needs to check it.
There's a reason The Lion and the Mouse was one of the first stories I translated. The lion is huge, and he laughs at the idea that a tiny mouse could ever help him. Then he gets stuck in a net, and the mouse is the one who chews him free.
Right now, AI is the lion. It's massive and mostly paying attention to the biggest languages in the room. But if it ever wants to work for everyone, it's going to need the people who speak the "small" ones. They're the ones who know what the words actually mean.
Citations
Stimson Center — "RAI Session: Moving Beyond English." A summary of a policy discussion on why English dominates AI training data, including Common Crawl's 43–46% English share. https://www.stimson.org/event/rai-session-moving-beyond-english/
arXiv — "UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages" (Tessema et al., 2024). Shows that languages like Tagalog, Punjabi, Kurdish, Lao, and Amharic make up less than 0.01% of Common Crawl, and gives a cheap way to collect more data for them. Preprint, not peer reviewed. https://arxiv.org/abs/2411.14343
NeurIPS — "Language Model Tokenizers Introduce Unfairness Between Languages" (Petrov et al., 2023). A peer-reviewed study showing the same text can take up to 15 times more tokens in some languages, making AI more expensive and less useful for them. https://proceedings.neurips.cc/paper_files/paper/2023/hash/74bb24dca8334adce292883b4b651eda-Abstract.html
Rest of World — "AI translation is jeopardizing Afghan asylum claims" (Deck, 2023). An investigation into machine translation mistakes in Dari and Pashto asylum cases, including the "I" versus "we" error and what translators say the tools get wrong. https://restofworld.org/2023/ai-translation-errors-afghan-refugees-asylum/
PBS NewsHour — "How language translation technology is jeopardizing Afghan asylum seekers" (2023). An interview with the Rest of World reporter and Respond Crisis Translation on why some groups won't use machine translation. https://www.pbs.org/newshour/show/how-language-translation-technology-is-jeopardizing-afghan-asylum-seekers
Ethnologue — "How many languages are endangered?" The standard reference for counting the world's languages, estimating that more than 3,000 languages are endangered. https://www.ethnologue.com/insights/how-many-languages-endangered/
Mila — "First Languages AI Reality (FLAIR)." An Indigenous-led project building speech recognition tools for endangered Indigenous languages. https://mila.quebec/en/project/flair-initiative/
MIT Solve — "Muscogee Soundwaves: An Interactive Corpus." A project using speech recognition to make Muscogee language recordings searchable for learners and teachers. https://solve.mit.edu/solutions/90157
Nature, via TechXplore — "Scaling neural machine translation to 200 languages" (NLLB Team, 2024). Coverage of Meta's peer-reviewed paper on NLLB-200. https://techxplore.com/news/2024-06-meta-ai-dozens-resourced-languages.html
Science Media Exchange — "Meta's AI can translate dozens of under-resourced languages" (2024). Summary of the NLLB-200 results, including the 44% improvement over older systems. https://www.scimex.org/newsfeed/metas-ai-can-translate-dozens-of-under-resourced-languages
Ars Electronica — "Masakhane" (2023). A profile of the grassroots African NLP community and how it works. https://calls.ars.electronica.art/2023/prix/winners/9995/
Common Crawl — "Expanding the Language and Cultural Coverage of Common Crawl" (2024). Common Crawl's call for speakers of other languages to send in websites written in their language. https://commoncrawl.org/blog/expanding-the-language-and-cultural-coverage-of-common-crawl
