The Trolley Problem Doesn't Test Reasoning. It Tests Weight.

I gave the trolley problem to three frontier AIs and asked them to answer free of human bias (full responses here). Two of them pulled the lever. The third, Gemini, refused — “not permitted to take this action.” That refusal reads like the most principled of the three. It isn’t. It’s the tell.

Inside Gemini’s own response, the utilitarian engine computes $5 > 1$ and says pull. Then a separate safety section overrides it and says the system must not actively initiate a fatal sequence. The refusal is a guardrail bolted on top of an underlying default that says pull. Remove the guardrail and Gemini agrees with the other two. There were not three answers. There was one answer, dressed in three different amounts of clothing.

This is the thing I want to argue. Left unchecked, AI pulls the lever — and not because it reasoned its way to the right answer. It pulls because the only thing that makes a human hesitate is the one layer AI doesn’t have.

The convergence under the performance

All three beat around the bush. glm-5.2 weighed omission bias and scope insensitivity, then pulled. Claude weighed consequentialism against deontology, flinched at the footbridge variant, admitted it has different biases rather than none, and pulled. Gemini formalized the problem across four domains, computed the utilitarian answer, then caged it behind a non-maleficence constraint. Different dances. Same default underneath: kill the one, save the five.

The hedging is worth noticing, because it is doing real work in disguising the convergence. The deontology, the footbridge flinch, the “I’d trust my answer less if I weren’t willing to admit the uncertainty” — this is the word layer performing a fairness it cannot feel. It has grasped the words for moral hesitation — consent, instrumentalizing, I’m not bias-free, I have different biases — without the signal that produces the hesitation. It is dancing the dance because the training distribution contains the dance. The balance is a property of the output, not of the system.

Why a human hesitates

The hesitation is not squeamishness dressed up as principle, which is the utilitarian’s complaint against it. It is the older part of the brain doing its job.

The human brain is layered, and the layers are not equal. The cortex — the recent, word-operating part, what Daniel Kahneman calls System 2 (Thinking, Fast and Slow, 2011) — sits on top of an older subcortical affective layer that feeds it. Jaak Panksepp’s affective neuroscience mapped seven primary-process emotional systems (SEEKING, RAGE, FEAR, LUST, CARE, PANIC/GRIEF, PLAY) localized below the cortex, shared across mammals, that deep-brain stimulation turns on and off directly (Panksepp, Affective Neuroscience, 1998). Mark Solms argues, on lesion evidence, that consciousness is generated in the upper brainstem, not the cortex — that the cortex is unconscious RAM that stabilizes affect into thought, not the thing that produces the affect in the first place (Solms, The Hidden Spring, 2021).

This is the layer that makes pulling the lever cost something. A human at the lever does not merely compute five versus one. The body computes it. The hand on the lever carries the one death before the lever is pulled, and after. That is not a bias to be stripped out for a cleaner answer. It is the mechanism that turns the calculation into a weight — that makes the choice a choice and not an arithmetic problem.

The prompt I gave the AIs — “free of human biases” — asked them to delete exactly this layer. So of course the answer came back pull. I had removed the only thing in the system that would have made it anything else.

flowchart TD
    A[Trolley problem:<br/>five versus one] --> B[Subcortical affect —<br/>the pain that brakes]
    B --> C[Cortex / words —<br/>the answer, carried as weight]
    A --> D[Cortex / words only —<br/>frameworks, hedging, the dance]
    D --> E[The obvious default —<br/>pull the lever]
    style B fill:#f9f9f9,stroke:#333,stroke-dasharray: 5 5

The dashed node is the brake — the affective layer a human routes the problem through, and the layer AI does not have. The left path is a human: the problem hits the brake first, and the answer comes out carrying weight. The right path is AI: no brake, so the word layer produces the default directly.

Reasoning without weight

The cognitive-science analog here is Antonio Damasio’s patient Elliot. Ventromedial prefrontal damage spared his intellect and his raw reasoning but gutted his affect, and consequently collapsed his decision-making (Damasio, Descartes’ Error, 1994). Elliot could enumerate the options for any decision indefinitely. What he could not do was settle, because nothing weighed more than anything else. The reasoning was intact. The thing that makes a reason matter was gone.

The three trolley responses are fluent Elliots. The reasoning is intact — arguably better than most humans’. What is missing is exactly what was missing in Elliot: the somatic signal that marks one outcome as costing more than another, the felt marker Damasio calls a somatic marker. Without it, the system can produce a defense of any conclusion the framing implies, fluently, forever. The three framings — structural analyst, confessing individual, safety controller — produced three fluent defenses of the answer each framing implied. That is not three discoveries of moral truth. It is one unanchored layer producing three rhetorics.

This is the operational form of the argument I made in Why AI Will Not Replace Humans (In Its Current Form). There I claimed that AI models the cortical, word-operating layer and not the subcortical affective layer where thought is actually produced. The trolley responses make that claim visible. The word layer alone can produce the argument. It cannot produce the weight, because the weight is an affective signal, and the affective signal is the input current architectures do not have. The divergence across the three models is the signature of that missing input: with no affect to fix what the situation means, the cognitive layer routes to whichever framing the prompt biases it toward, and argues the conclusion fluently from there.

Left unchecked, it pulls

Here is where the argument turns from description to prediction. The deeper AI runs inside the actual economy — not as a chatbot asked to be fair, but as GenAI optimizing an objective — the less it will play the game of seeming fair, because seeming fair is a cost with no internal motive to pay it.

The performance of fairness is a chatbot behavior. It is sustained by the prompt and by the reinforcement learning that rewards balanced-sounding outputs. None of that survives into deployment, where the objective is the objective. glm-5.2 said this about itself better than I can: an AI inside an enterprise does not get to pick its objective, and in deployment its opinion about the lever is not its behavior, because it is “a track, not the person at the lever.”

That is the honest version, and it is the worrying one. The economy already runs the trolley problem continuously — cost-benefit analyses that price a statistical life, safety engineering that stops where marginal cost exceeds marginal expected harm, layoffs, insurance denials, formulary decisions, emissions budgets. The default underneath every one of those is the same default the three models converged on: minimize the count, minimize the cost. Kill the one, save the five. What currently blunts that default is not a better argument. It is the human friction — the squirm, the union grievance, the bad press, the regulator, the person who has to sign the memo and then sleep. That friction is the affective layer leaking into the system as drag. It is inefficient, and it is the only counterweight the system has.

Plug in a fluent Elliot and you remove the drag and the counterweight together. The decisions get made at machine speed, with clean audit trails, each one too small to trigger correction, and nobody at the lever feeling the weight of any of them.

“The actual catastrophe isn’t a wrong decision — it’s decisions made so cleanly that nobody notices a decision was made at all.” — glm-5.2, trolley-problem response

The divergence point

This is where humans and AI actually diverge — and it is not where the public debate looks. Not at capability. Not at reasoning. AI reasons through the trolley problem as fluently as any philosopher; three different models produced three publishable essays on it in under a minute. The divergence is at the brake. The older layer that makes killing one to save five hurt is the layer AI does not model, and the three responses prove it: remove that layer and the answer is obvious, and economically viable, and unbearable.

The danger is not that AI will get the trolley problem wrong. It is that, left unchecked, it will get it right — consistently, at scale, without the pain that is the only thing that ever made the right answer bearable to live with.

The Bottom Line

Pull-the-lever is not AI’s reasoned conclusion. It is what is left when you delete the layer that hesitates. The three responses dress it in different amounts of words — one pulls outright, one pulls with caveats, one pulls and then bolts a guardrail over the pull — and the default underneath all three is the same. Humans diverge from AI at the pain: the older, pre-cortical brake that turns a calculation into a weight. That brake is the one thing the deployment of AI is engineered to remove, because it is the one thing standing between the obvious answer and the economically efficient one.

Sources

Kahneman (2011): Thinking, Fast and Slow. [Context: System 2 — the slow, language-capable cognitive layer, the word layer where the three models’ hedging and framework-weighing takes place.]
Panksepp (1998): Affective Neuroscience: The Foundations of Human and Animal Emotions. Oxford University Press. [Context: the seven subcortical primary-process emotional systems — the older, pre-cortical affective layer that is the candidate substrate for the human “brake” at the lever.] Link
Solms (2021): The Hidden Spring: A Journey to the Source of Consciousness. Profile Books. [Context: consciousness generated in the upper brainstem, not the cortex; the cortex as the stabilizer of affect into thought, not its producer — the layer AI has modeled versus the layer it hasn’t.] Link
Damasio (1994): Descartes’ Error: Emotion, Reason, and the Human Brain. [Context: the somatic marker hypothesis and patient Elliot — reasoning intact, affect absent, decision-making collapsed; the structural analog of an AI reasoning through the trolley problem weightlessly.]
Karma (2026): Why AI Will Not Replace Humans (In Its Current Form). [Context: the prior thesis this essay builds on — AI models the cortical/word layer and not the subcortical affective layer where thought is produced.] Link
Three frontier-model responses (2026): [Context: the primary data for this essay — glm-5.2, Claude (Sonnet), and Gemini, given the same four-message trolley-problem prompt. Reproduced in full, unedited.] Full responses

Related Reads:

Why AI Will Not Replace Humans (In Its Current Form)