My househelps phone rang the other day while she was in the kitchen, and she put it on speaker the way she always does.
I half-listened from the next room as a voice told her, in Hindi, that her recharge plan was about to run out.
She talked it through with the voice, asked it which plan would last her the month, asked again to be sure, and thanked it twice before hanging up.
When I told her she'd spent those five minutes talking to a program, she gave me a polite look, but it felt like I'd misunderstood or said something wrong.
As far as she's concerned, she spoke to a person that was kind to her. But that trust in the voice concerned me and that’s what I want to write about today: AI can be extremely deceptive for vernacular users.
You and I are careful with AI.
We fact-check it and catch it making things up, and roll our eyes at the people who don't.
By the time something good arrived, we already knew in our gut that the machine is wrong until it proves otherwise.
She skipped that whole decade.
The first machine that ever spoke to her did it by voice, in the language she thinks in, and it was fluent and warm from the very start.
Nothing it did was ever clumsy enough to break the spell, so she never learned to doubt it.
A lot of what I call judgement is just damage we’ve learned to be proud of.
It's tempted to think of her as a one-off case but the numbers say otherwise.
India became Meta AI's largest market in the world within months of it switching on, because it lives inside WhatsApp, where the country already spends its day.
We are also the 2nd largest market in the world for ChatGPT, with around 100 million people using it every week, and the largest student base of any country on it.
We have watched this happen before.
The country barely bothered with the landline and went straight to the mobile, then did the same with money, skipping the credit card and landing on UPI, which now runs more than 8 in 10 of our digital payments and crossed 21 billion transactions in December alone.
Each time, the people without the old habit moved faster than the ones who already had it because they had nothing to unlearn.
This would be a purely happy story if the machine had earned the trust.
The whole country is now building for her.
Sarvam, the startup the government picked to build India's own model, launched 2 of them in February, and the government's own Bhashini system already works across 22 languages.
Razorpay and Swiggy are wiring up shopping you do by talking, in Hindi.
At Mumbai Tech Week, Urban Company showed a demo where a technician standing in a customer's house with a dead water purifier just talked the fault through with an AI in Hindi and got walked straight to the broken part. (and what kinda triggered me to write this today)
The people building the biggest systems already understand her better than I do.
She is the market now.
And the AI is weakest in exactly the languages she trusts it in most.
A model makes things up when it sees too little to know better, and the internet holds far less writing in Hindi or Tamil or Marathi than in English.
The safety training, that teaches it to refuse a dangerous answer, is built mostly in English too.
Some researchers took dangerous requests that a leading model refused almost every time in English, translated them into a few of the world's smaller languages, and the refusals fell apart, so a model that said no 99 times out of 100 went along with about 79 of them once the words were no longer in English.
So the person least able to feel when the machine is confidently wrong is using the version of it most likely to be confidently wrong.
She used to hand me the phone whenever a call slipped into English, and she doesn't anymore, because the voices have started coming to her in her own language, with all the patience in the world.
Maybe the scar protects me, or it just holds me back while she runs ahead.
I can't tell which so I'll hand it to you.
Think of the one person in your life who trusts AI completely, in their own language, and has never once stopped to doubt it.
Reply and tell me which one they are: the one who has got this right, or the one it's going to hurt first.
Nandini