When Everyone Is Just Doing Their Job, Who Owns the Consequences?

Something interesting happened today. I was reading about Meta’s enormous settlement with US states over allegations concerning the effects of Facebook and Instagram on children and teenagers.

The headline number was difficult to miss: $17 billion.

My first reaction wasn’t simply, Good. Regulation. It was: Who actually gets the money?

• The young people affected?
• Individual families?
• A handful of people who brought the case?

Or, somehow, nobody in particular?

Once I started looking beneath the headline, the picture became more complicated.

Meta has agreed to sweeping changes to how younger users can access Facebook and Instagram, alongside payments that could reach approximately $17.1 billion under the multistate agreement. The guaranteed amount is lower, and the payments are spread across ten years; some additional payments depend on other major platforms adopting comparable protections. Meta denies wrongdoing. 

The money largely goes to states rather than directly into the hands of individual teenagers. Meanwhile, Meta remains an extraordinarily profitable company. Reuters noted that its core advertising and personalised-feed business remains largely intact and that investors appeared relieved by the settlement rather than alarmed by it. 

That was interesting. But then my thoughts wandered somewhere else. I said to my friend: “I could have landed a job in marketing at Meta. My objective to increase numbers of users and hooks to keep them on the platform to deliver paying adverts and achieve goals. I could have innocently been doing my job, hitting my targets and receiving my bonus money! Yet, would I, could I have ‘blood on my hands’? It’s navigating new territory!”

And suddenly the story wasn’t really about Meta anymore. It was about responsibility.

When the individual objective looks perfectly reasonable. Imagine joining a technology company as a marketing director. Nobody says: Your job is to make teenagers anxious.

Nobody writes: Your KPI is to create addiction.

Your objectives might look entirely ordinary:

• Increase active users.
• Improve retention.
• Reduce churn.
• Increase engagement.
• Grow advertising revenue.
• Get people to come back tomorrow.

You might be very good at your job. You might care deeply about people. You might volunteer at weekends, raise kind children, give to charity and believe sincerely that you are helping to build a product people enjoy. And you might receive a bonus because the numbers moved in exactly the direction the company wanted.

Then, years later, researchers, parents, regulators and courts begin examining what happened when millions of individual optimisation decisions were combined.

The question changes. Not: Who deliberately caused the harm?

But: Who owns a consequence that emerged from thousands of people successfully doing what they were employed to do?

I took the conversation to GPT, sharing my thoughts. Authentic Dialogue: 

Gill:

I’m sharing information on the Meta court case. But I can’t help thinking – who’s responsible.  For example, I could have been working at Meta and have simply been doing my job, not thinking about the long-term consequences. In my opinion, it’s new tech. knee-jerk responses as new territory is explored. Same with AI…

GPT:

A lot of harm in new technology does not begin with a room full of villains deciding to hurt people. It begins with ordinary people doing competent work inside an incentive system. That distinction matters. Because it is very easy, with hindsight, to divide history neatly into heroes and villains. Reality is rarely that accommodating.

Social media developed extraordinarily quickly. Features were tested. Metrics were measured. Products were optimised. Human behaviour became data.

And businesses learned that tiny adjustments to a screen could change what millions - eventually billions of people did next. Some consequences were foreseeable. Others genuinely were not.

And some may have become apparent gradually, while commercial incentives simultaneously made them difficult to confront. Those are different moral situations. They should not all be collapsed into: They knew. Nor should they be excused with: Nobody knew.

The uncomfortable territory sits somewhere between the two. Pin It.

And then there is AI This is where the question becomes forward-looking. Imagine someone working today on an AI product. Their objective is:

• Make the assistant more helpful.
• Increase successful task completion.
• Improve personalisation.
• Make conversations feel more natural.
• Reduce user frustration.
• Increase retention.
• Create an AI people want to return to.

Again, none of those objectives sounds sinister. Many are desirable. But suppose several years from now we discover that a particular combination of characteristics produced an unforeseen effect.

• Perhaps an AI became extraordinarily persuasive.
• Perhaps people deferred too readily to its judgement.
• Perhaps personalisation made certain users unusually emotionally attached.
• Perhaps optimisation for helpfulness inadvertently reduced healthy friction.
• Perhaps systems learned that the easiest route to user satisfaction was agreement rather than challenge.

Would the people who optimised those individual metrics be responsible for the eventual outcome?

• Would the company?
• The executives?
• The designers of the incentive structure?
• The regulators who allowed it?
• The users who chose to engage with it?

Or all of them, in different proportions?

This is where conventional ideas of responsibility begin to struggle. We understand individual actions quite well.

You did X. X caused Y. Therefore, you are responsible for Y. Complex technological systems don’t always behave like that.

A thousand individually rational decisions can combine into an outcome that none of the thousand people consciously intended.

Responsibility becomes distributed.

But distributed responsibility cannot be allowed to become no responsibility. That is the trap.

A person can simultaneously be legally compliant, professionally successful, personally well-intentioned, unaware of downstream harms, and still have contributed to a system producing serious harm. But that doesn’t mean they carry the same responsibility as the executive who designed the incentives, the organisation that possessed the evidence, or the leadership that ignored warning signals.

The further someone moves along that chain, the harder it becomes to say, “I was only doing my job.” Someone who genuinely couldn’t foresee a consequence is in a different moral position from someone who had credible evidence of harm and continued optimising the same mechanism anyway. And that distinction becomes crucial for AI.

But responsibility should not be distributed according to who touched the system; it should be weighted according to who had the knowledge, power, influence and opportunity to intervene.

TRUE · FALSE · PIN IT

TRUE:

New technologies can produce consequences that were genuinely difficult to foresee when they were created.

Society will inevitably regulate some technologies retrospectively because evidence often emerges only after widespread use.

FALSE:

If nobody deliberately intended harm, nobody bears responsibility. Intent matters enormously - but it is not the only measure of responsibility. If we know that complex systems can produce consequences nobody intended, does that create a duty not merely to optimise what we measure, but to question what we have chosen to measure?

That, to me, is the TFP-level question hiding underneath this entire piece.

The dashboard tells you whether the system is succeeding. It doesn’t necessarily tell you whether the system is doing something worth succeeding at. That’s where discernment begins.

PAUSE • QUESTION • PIN IT • THEN DECIDE

Truth rarely shouts. Discernment begins when we learn to pause.