Episode 13 — Who Decides What's "Safe" AI? Ethics, Trust & the Global Language Divide

Introducing Dr. Sarah Luger

Sunil Baliga: Token Drop, Friday afternoon — and we're here with Dr. Sarah Luger. I met Sarah a couple of weeks ago through a mutual acquaintance, and I'm really glad I did — talking with her, I could tell right away she had a deep well of knowledge and interesting takes on AI, and I thought she'd be a great fit for the podcast. She graciously agreed to join us.

A bit of background: Sarah has a PhD from the University of Edinburgh, and she's an LLM AI safety and evaluation expert as well as a generative AI research director. One of her more notable projects was working at IBM on Watson for the Jeopardy Challenge. She's well published, and it's been a real pleasure getting to know her over the past couple of weeks.

I wanted to start with something I've been thinking about: we hear a lot of terms in AI — trust, safety, ethics — that carry emotional or human connotations. But LLMs don't have emotions; they're algorithms. So in the context of AI, what do these terms actually mean?

From ethics to responsibility to trust: how the vocabulary evolved

Dr. Sarah Luger: Thank you for the warm welcome — I'm looking forward to this. The naming conventions around AI trust, ethics, and safety are genuinely important to unpack, because before ChatGPT and the broader LLM wave hit, the conversation looked different. There was some discussion of AGI, especially in Silicon Valley, and around AI ethics specifically.

"Ethics" is a term more associated with philosophy than the hard sciences — and I'd argue it's a core part of our humanity, about what we believe is right and wrong. But these days, the term tends to be read as more positional: "these are my ethics, and that's someone else's ethics." That's part of why the field shifted from "ethics" to "responsible AI" — an effort to create a broader, more shared umbrella for what we collectively consider the correct approach.

From there, the language shifted again toward "trust" and "transparency" — building systems whose output we can actually trust. There's a through-line connecting all of these terms. What's interesting lately is that the sheer power of generative AI has pushed people to reframe failures and errors in AI systems as fundamentally a safety issue — closer to a cybersecurity-level concern within a company. Years ago, "AI ethics" might have been a nice-to-have, something a values-driven European company invested in to align with its customers in a specific way. Today, the posture across the board is closer to: we want secure AI, we want safe AI.

Sunil: So what do you mean specifically by safety and security in this context?

Sarah: One way to frame safety is: by using this product, can I be exposed to harm that's unreasonable? "Unreasonable" is admittedly a slippery word we could spend a lot of time on. A friend of mine uses an analogy I like for how AI products get sold globally: imagine buying a car in the United States — full functionality, working seatbelt, working brakes. That same car, sold in West Africa, might have a seatbelt that doesn't work, because the product wasn't designed with that market's language or context in mind. Is there a real functional difference that makes the product unsafe in one market versus another?

Every nation curates the internet differently, with different standards. Right now, "AI safety" is often framed around whether these large language model systems can produce harms that would be considered unreasonable relative to what you'd already find on the open web. There's a real conversation to be had about what's already findable in a library or on parts of the web versus what LLMs make available — but there's something genuinely different about the just-in-time delivery of this information. It feels more powerful simply because it's now at our fingertips in a new way.

Does that help clarify?

The cultural problem: whose definition of "safe" are we using?

Sajjad Khazipura: It does — and it raises something for me. You touched on ethics, safety, and trust, but these words mean different things across different cultures and countries. There's no standard, agreed-upon definition. And we're now seeing models emerge from many different countries — I was just looking at some Open Router statistics showing that an increasing share of U.S. business traffic is being directed toward models built outside the United States. Those models may or may not have gone through the same verification, validation, or certification processes we'd expect domestically. What does that mean for us?

Sarah: I think it's actually a great opportunity — a very capitalist way of framing it, but it's a real opportunity to differentiate and build products that genuinely serve what different customers want. Your point about community is an important one: what counts as a "hazard" in the West isn't universal. At ML Commons, I've worked on the AILuminate evaluation — a benchmark that lets model builders (and really, anyone) evaluate how their model performs against twelve standard hazard categories that Meta introduced a few years back. Those categories are themselves rooted in a Western, U.S.-based frame of reference. Different communities have different concerns.

We're all based here in the U.S., but we sell products globally. Does that mean we should default to the most conservative approach — the most aggressive safety flags across the board? That could be one path. When I worked at Orange, the French telco, GDPR was obviously central to our work, but Orange also operated heavily across West and North Africa and the Middle East, in countries that didn't yet have GDPR-style laws. Our approach there was to adhere to the most protective, most privacy-enforcing standard, regardless of where the customer was located.

Having individual nations — the EU, California, other states and countries — weigh in and say "here's what we're hearing from our constituents, here's what matters to us" is a genuinely important step, even though we'll disagree on plenty. What makes recent AI so powerful is that it uses language in an extremely naturalistic way, and communication is what differentiates us as humans. The data LLMs are trained on wasn't, for the most part, generated specifically for LLMs — it was people talking about their meal in a Yelp review, ranting about a Marvel movie on Reddit. Beautifully, ordinarily human. So we're going to see a lot of that humanity reflected back in generated language — but it's also a chance to step back and ask: what are we actually trying to get out of communication, and what community norms do we want these systems to adhere to?

Low-resource languages: why most AI is built for the Global North

Sajjad: And a lot of the training content these models rely on skews heavily Western.

Sarah: Yes!

Sajjad: You've actually written a paper on data infrastructure for low-resource languages and the challenges there. Can you speak to that?

Sarah: I can — though I want to push back gently on the framing of "optimizing" for all these different trust and safety regulations, which is a very Bay Area way to think about it. It's less about optimizing and more about surveying carefully: what works best for us, understanding that what works for us inevitably bleeds into other communities. We're not operating in a vacuum.

This connects back to the car analogy and the English-centric nature of a lot of these systems. If a system is built primarily in English, with agile-methodology user personas that are themselves Western-centric, it's not surprising that it doesn't work well for everyone.

There are, broadly, two philosophies here — and this might generate some genuinely contentious feedback. Data is the fuel that builds large language models. When a language is primarily spoken rather than written — an oral tradition rather than a text tradition — you inherently have less data to work with, at least under today's largely text-to-text (and some text-to-speech) paradigm; we still have comparatively little speech-to-text data for these languages. Sam has worked in this space at Dragon and understands that some of the foundational steps — phoneme modeling and so on — aren't yet automatic for many of these languages, though the technical tools are advancing quickly.

When you have oral traditions, you often don't have a standardized orthography — a standard written form — for the language. And a Western keyboard layout may not even align well with that orthography. There are real practical challenges: a physical keyboard assumes a laptop, but what about people using phones? What does real global language use actually look like? It's code-switching, it's neologisms — new words blended from multiple languages and cultures. Reddit is a good example of this in English: it captures the real vernacular of younger, tech-savvy — and, globally, often wealthier — people who have the technology to record their thoughts and communicate that way.

Gathering more data is the first, non-contentious part of the answer, and there's a lot of good work happening there — I support several researchers doing excellent work in this space, and I'm fortunate to be part of an ecosystem that's gained real traction in recent years.

Data sovereignty: should communities be able to opt out?

Sarah: The other side of the low-resource language question is this: in the West especially, we've largely accepted that "the horse has left the barn" — AI is just here, your data is out there. Some communities aren't comfortable with that. They view their language as sacred, with deep religious and historical significance attached to how it's controlled and shared.

I want to be clear that, as someone who isn't from one of these communities — though I grew up around many people who are — I respect nuances I'll never fully understand, because I come from a dominant, historically colonial language. The language I speak best, besides Python, is English.

If the "solution" to turning a low-resource language into a high-resource one is simply generating more data in that language, there's a real tension worth naming. Right now, if people in a low-resource-language community speak English or French, they can access all of the existing software engineering tools and build great products — even generational wealth — in those dominant languages. But imagine if they could do that in their own language, without having to hop to another one first. That would be fantastic — it's not my place, or anyone's, to gatekeep someone driven and capable from using these tools. I believe access is a right.

At the same time, my relationship to language is very different from that of some community members I've met, and I believe there should be real mechanisms for Indigenous and First Nations communities to "de-platform" their own data if they choose to — meaning, if there's less of a specific language represented in web crawls (the kind that tools like Common Crawl gather), it becomes harder to identify, harder to model, and in a sense harder to find in order to ask, "do you want this removed?" The flip side, as Sajjad's point suggests, is that this Western-centric approach can feel like it's steamrolling other cultures — the tension between reducing culture and language to data, versus using these tools to let people express themselves as they choose, is real and unresolved.

What does the public actually need to understand about AI reliability?

Sam Pooni: This kind of rigor is something Sajjad's touched on in past episodes too — most internet data is fundamentally English-language in nature, though we're now seeing serious models coming out of China and elsewhere. The question we keep coming back to is: are these non-English-centric models really able to handle the same nuance as an English-based model? Generally, not yet. So when it comes to low-resource-language models specifically, what do you think the public most needs to understand about AI reliability — something the industry isn't saying plainly enough?

Sarah: This might be more Damon's territory, but at some level, our relationship with AI products and companies comes down to brand marketing and brand trust. When I make a purchasing decision, or decide which LLM to use, that's rooted in earned reputation — just like with anything else.

[Sarah briefly pauses due to noise outside — an unrelated interruption.]

Can low-resource communities build their own models?

Damon Miller: I have a question, given your work with the Global South and AILuminate: is the path to democratizing AI — while preserving the culture of these lower-resourced, less-represented languages — for those communities to develop their own open-source, sovereign models? And if so, there's a real compute cost to training and running those models. Is that economically viable on its own, or does it require something like a UN-style coalition to help fund these efforts?

Sarah: That's a great question. Back in 2017, I read an Accenture report on AI adoption — this was pre-ChatGPT, before the mainstream normalization of LLM products. It found that the tax money, corporate money, and nonprofit money flowing into AI across different geographic regions is deeply unbalanced — and the wealth being generated is significant enough that if this trend continues, much of the world may not even be able to afford the AI products being sold to them, given the scale of the economic and technical head start certain companies and countries have.

I think that particular ship may have already sailed. I do wonder whether groups of nations need to come together — maybe through the UN, maybe through regional coalitions — to build sovereign, open-source models and figure out what matters most to them. I don't think there's one blanket solution for every community. Mali is an interesting example: a very impoverished nation, a former French colony, with one of the highest birth rates in the world — nearly six children per woman on average. It's part of the Global South and faces real challenges, but it's also leaned into aspects of independence, including investing in a ministry of language, actively trying to move past prior generations' reliance on French as the vehicular language for economic advancement, and instead investing in teaching and standardizing local Indigenous, low-resource languages. The idea is: build more language tools, which support better data collection, which can eventually support building your own models. I don't think there's a single blanket answer, Damon, but it's a genuinely interesting challenge.

Damon: It seems like we're heading toward something like what's discussed in a forward-looking paper — I think it's sometimes referred to informally as "2031" — that describes a kind of duopoly between the U.S. (with its foundation models) and China (with its manufacturing and physical AI presence), with everyone else left choosing a partner because they can't afford to invest in compute independently. Its conclusion is essentially: if countries don't start partnering and investing in compute now, they'll be left behind by 2031.

Sarah: That resonates with a lot of the same concerns — and it's striking how prescient some of those ideas are. This isn't just about which companies are involved; it's about where they're based, who they're hiring, where they're sourcing power. We've all heard about the environmental costs — this isn't only about using these tools, it's about the infrastructure required to support the compute. Fresh water, for instance, is a genuinely underappreciated input to a lot of these systems.

One thing worth remembering in the U.S. context: many of the technology companies doing extremely well today are building on research that was originally government-funded, largely post–World War II — military-driven communications infrastructure that, in the postwar era, we treated almost like a peacetime arms race, with enormous public investment. A lot of my grandparents' tax dollars, for example, helped build the underlying internet infrastructure that companies like Google and Amazon have since built enormous businesses on top of.

Damon: GPS being another good example.

Sarah: Exactly — I know we're not supposed to call out specific companies, but the big ones. That history reflects the long-term wealth of our nation and a deliberate choice to invest early. It's also worth asking: why are some of our translation systems noticeably better at Russian and Arabic than at other languages? There are clear geopolitical reasons behind that. But that doesn't mean the current imbalance has to persist — our customers are everywhere. What happens if this kind of asymmetry calcifies, and customers simply can't afford the products because the ecosystems that built them are so expensive to sustain?

Moderation models, benchmarks, and the case for a public leaderboard

Sajjad: Shifting topics slightly — a lot of companies are now building moderation models, and a few are even available on Hugging Face. But not all of them are being benchmarked against something like ML Commons' standards. They might be running internal evaluations, but we haven't seen public evidence of scores — there's no visible leaderboard. Can you speak to the need for a public leaderboard, and for encouraging companies to actually run their models against these benchmarks?

Damon: And maybe first — how would you define a "moderation model" for listeners?

Sarah: As a fan of science generally, I really like the concept of a leaderboard — and I think one of the best examples ever was the Netflix Prize, about twenty years ago. For anyone unfamiliar: Netflix offered a million-dollar bounty to whoever could improve their recommendation algorithm's prediction accuracy by 10%. It became a genuinely interesting competition — a message board sprang up, and for a while nobody was getting close to the threshold. Eventually people realized the rules didn't require an individual entrant — they allowed teams. The people who formed teams ultimately won. I like that story because it reflects the real, less mythologized way great ideas actually come together — through teamwork, rather than the lone-hero narrative we tend to default to.

A leaderboard, done well, genuinely pushes science forward: can I get this result, and is it replicable? Hugging Face supporting that kind of ecosystem is great. That said, I think we've also over-indexed on leaderboards recently — now everything has one. I want to support the underlying scientific method — hypothesis, experiment, analysis, repeat — but it's worth thinking carefully about who's actually using these leaderboards, and how that's shaping real decisions.

To your original question, Sajjad: are these unpublished internal scores good, or should we be concerned that some companies aren't opening themselves up to public benchmarking?

Sajjad: Right — my observation, more than a firm point, is that some companies appear to be using ML Commons benchmarks internally, but they're not publishing where they stand publicly. I haven't seen a public leaderboard showing where a given company or model stands. If you're serious about building trust, safety, and ethics into your product, it seems prudent to support a leaderboard that lets consumers compare options and say, "this is what I want."

Sarah: Kind of like a Consumer Reports for AI models.

Sunil: Something like that.

Sarah: Something accessible, yes. I think buy-in through a genuine consortium — with real investment, similar to how the W3C or ISO standards work — around standardization, evaluation, and benchmarking is valuable. NIST, the U.S. government's standards and measurement body, has done successful work in this space historically — the IBM Watson Jeopardy Challenge, for instance, grew out of some earlier NIST benchmarking challenges. Chess fans might also think of the Deep Blue challenges — a lot of these public, high-visibility AI milestones have roots in government-sponsored competitions.

I'm generally in favor of these efforts being consortium-based — academic institutions and major companies both involved. I'd agree that many, if not most, companies are using ML Commons' AILuminate and other standards internally, as part of their development cycle — a way to sanity-check whether recent changes and bug fixes are actually improving output. But I also understand this is a competitive landscape, and not every company sees clear value in publishing those results. The people actually using these benchmarks aren't all making large-scale purchasing decisions off of them — some buyers simply aren't prioritizing trust and safety in the first place.

Who should be accountable for grading AI models?

Sam: So, Sarah, on that note — who should actually hold the grading pencil here? The lab itself, a standards body, or a third party with no commercial stake? What's the failure mode for each?

Sunil: Sam, keep it quick if you can — we're running short on time.

Sam: Sure — same question: who should hold the grading pencil, and what goes wrong with each option?

Sarah: I believe strongly in human-in-the-loop evaluation, because these are ultimately products built for us. We get very excited about AGI — "woohoo, AGI!" — but these systems exist for people. I think we sometimes forget that. This is fundamentally about communication, and communication is inherently flawed — I'm thinking something, feeling something, using all my senses, and what comes out is maybe a 90%-accurate version of what I actually meant in whatever language I'm speaking — and somehow, we still understand each other.

So who should be in charge? I don't think it should be one person — it should be a group of people with different perspectives, each capable of roughly that same "90% quality" communication, because that diversity is what helps you establish real inter-annotator agreement. A human-in-the-loop group like that also helps surface the right questions in the first place — whose voices aren't in the room? One of the real challenges in evaluating something like hate speech or defamation is that some of the relevant terms are deeply offensive to specific groups, while many people outside those groups simply aren't aware those terms exist — which, in one sense, is a good thing; less awareness of hateful language circulating is good. But it also means those terms remain extremely harmful to the people they target, and evaluators need to maintain a genuinely wide, constantly updated awareness of harmful language, because language keeps changing.

Sunil: Language changes everything.

Sarah: Exactly. So: a consortium made up of many people, with different perspectives from different communities.

Wrap-up

Sam: Thank you — this was great.

Sunil: Thanks so much to Dr. Sarah Luger for joining us — genuinely thought-provoking conversation. And I should mention, as I usually do at the start: all opinions expressed here are those of the individuals themselves, not necessarily of any company they work for. Thank you, Sarah — if you can stick around for just a second, we'll chat after we wrap here.

Sajjad: Thank you.