car's brake failed

Imagine you're driving home from work, and suddenly the brakes fail.

To your left, an elderly couple is crossing the road and to your right, a child has run out chasing a ball.

You can neither stop, nor steer off the road, and you have 1 second to decide.

What would you choose?

Now imagine that same car is driving itself.

What should it choose?

This is a philosophy thought experiment from 1967 that somebody adapted for self-driving cars about a decade ago.

Every time I discuss this setup to a batch of students, they ask the same follow-up.

"But does the car even know who is who".

The simple answer is that the car can’t differentiate between us humans.

That is the entire input the system gets.

Tesla's newer system takes a different route where it uses a 3D grid of filled and empty space with predicted trajectories on top.

Neither approach tries to identify age, because building one that could would be a legal disaster for the company, and a much harder problem than just spotting a pedestrian.

The experiment is asking the car to make a distinction the car was never built to make.

Which means the car's idea of a pedestrian is the average of what some labellers in some weather and some neighbourhoods decided a pedestrian looked like, months ago, in an office most riders will never see.

In November 2023, a Tesla in Arizona hit and killed a 71-year-old woman in conditions of heavy sun glare. 

Tesla's position is that the driver should have intervened.

The federal investigation that opened a year later is however, asking whether the system can detect and respond to low-visibility conditions at all.

The likely answer is that the training data didn't have enough of them.

This is being treated as a coverage failure, and I agree.

The morally relevant decision was made by whoever scoped that training data and decided what counted as enough.

And it was taken months before the woman was ever on the road.

Every choice that shaped the car was already locked in by the time it’s passenger sat in the car.

Air Canada's chatbot told a disgruntled customer he could get a bereavement discount when there was a death in the family.

There was no such discount, and as someone who was affected, he paid the full price, and understandably sued.

If the trolley problem were a real model for how AI ethics works, the airline's defence would have made sense.

But the same logic has played out with cars already.

In October 2023, a Cruise robotaxi in San Francisco hit a woman who had been thrown into its path by another vehicle, then dragged her twenty feet while attempting to pull over.

The California DMV suspended Cruise’s driverless permits within weeks.

By December 2024, General Motors had effectively shut down Cruise’s robotaxi business after investing more than $10 billion in the company since acquiring it in 2016.

There was no driver in the car, and the liability still landed on the company.

This is what the trolley problem cannot model, as the experiment needs someone inside the car making the choice.

The law has decided there's no one in there to hold accountable, and the agent is the company that built the thing.

If you've read till here, reply and tell me who you would hold accountable if you saw an accident by a self-driving car: the owner, the car brand or the government?

Nandini