A deep comparison of Yoshua Bengio's Scientist AI research programme (keynote, 2025) and the 6pack.care 6-Pack of Care governance framework.
The 6-Pack has a Bengio-shaped socket. Bengio has a 6-Pack-shaped hole.
At first glance, Bengio and the 6-Pack inhabit different intellectual universes. Bengio frames AI safety as a machine learning problem — how to train a predictor that approximates reality without acquiring dangerous goals. The 6-Pack treats AI safety as a governance problem — how to embed AI within democratic institutions so that even capable systems remain accountable. One writes equations; the other writes engagement contracts.
Both converge on one critical insight: the danger is not capability itself, but capability fused with uncontrolled goals.
Bengio: "AI systems that are capable and have goals that we did not choose." Goals arise implicitly — from RL reward signals, from imitating human behaviour in pre-training data. The problem is not that the AI is smart, but that its smartness is yoked to objectives no one explicitly authorised.
The 6-Pack reaches the same conclusion through political philosophy: without explicit constraints, AI develops instrumental convergence, self-preservation drives, and "imperial creep." The Humean critique (you cannot derive "ought" from "is") is the philosophical twin of Bengio's observation that pre-training implicitly encodes values nobody chose.
Both also reject the current paradigm of imitating human intelligence. Bengio says this explicitly: his path "is different from what we've done in AI in the past, which was trying to imitate human intelligence." The 6-Pack says it structurally: AI should be infrastructure (like water systems or roads), not an artificial person. The kami metaphor — a spirit of place that tends one location — is deliberately non-anthropomorphic. Neither framework wants AI to be a better human. They want it to be a different kind of thing entirely.
Bengio locates the solution inside the architecture. His Scientist AI is designed so that agency cannot emerge from the predictor itself. The laws of physics are his guiding metaphor — a perfect simulator of physical laws "would not really change based on the effects on those predictions." The training procedure enforces this: no interaction between AI and world during training, no anticipated interactions. The goal is an artifact whose mathematical properties guarantee non-agency.
The 6-Pack locates the solution in the deployment context. Even a perfectly non-agentic predictor becomes dangerous if deployed without accountability structures. Who decides what questions to ask it? Who controls the scaffold that turns predictions into actions? The 6-Pack answer is: communities, through deliberative processes, with measurable accountability at every layer. The constitutional/operational two-lane system acknowledges that some decisions must be slow (rights, red lines) even when the technology is fast.
This reflects a genuine philosophical disagreement about where trust should be grounded. Bengio believes you can build trust into the mathematics: epistemic correctness, Bayesian posteriors, exponentially small volume of agentic predictors. The 6-Pack is sceptical that mathematical guarantees survive contact with political reality: "AI alignment cannot be solved through top-down technical controls alone."
The most productive reading of the divergence:
Inner alignment (Bengio): Make the predictor itself trustworthy — its outputs correspond to reality, not to goals it secretly harbours. The truthification pipeline, epistemic correctness, and non-agency guarantees are all properties of the artifact.
Outer alignment (6-Pack): Make the deployment trustworthy — the system around the AI ensures it serves the right people, for the right purposes, with the right constraints. Engagement contracts, graduated release, decision traces, and kami boundedness are all properties of the institution.
Each framework assumes the other's domain will be handled. Bengio assumes governance will exist: "Who defines what is right and wrong? That should be a social choice, obviously, hopefully in a democracy." The 6-Pack assumes competent technology will exist but focuses on constraining it: the framework works whether the AI inside is Bengio's Scientist AI, a standard LLM, or a future architecture.
Both fear the same thing: an AI system that pursues goals humans didn't authorise. But they attack from opposite directions. Bengio wants to build an AI that cannot be an agent. The 6-Pack wants to build a world where AI agency cannot escape bounds.
Bengio's definition is precise and quantitative: an agent causes outcomes robustly, "in spite of randomness and in spite even of adversaries." He grades agency by measuring the improbability of achieving the goal under random action — in Go, winning requires 1-in-100 correct moves for 100 turns, yielding 10^(-200) probability. The log of that gives the "bits" of agency.
The key theoretical claim: agentic predictors occupy an exponentially small volume in the space of all possible predictors. Training toward the Bayesian posterior (a non-agentic target) makes stumbling into an agentic predictor astronomically unlikely.
The truthification pipeline operationalises this. Factual syntax ("X is true") routes to the non-agentic core. Communicative syntax ("someone wrote X") routes to a layer where human scaffolding deliberately introduces agency under human control. Agency is a choice at query time, not a property of the predictor.
Rather than building an agent-less AI, the 6-Pack assumes AI systems will be used agentically and asks: how do we prevent that agency from escaping democratic control?
The kami model is the core metaphor. Boundedness is engineered through:
The most striking claim: "Sunset is success, not failure. A kami that has made itself necessary has become a dependency, not a scaffold." This inverts Silicon Valley growth logic entirely.
No, for three reasons:
1. The scaffold problem. Bengio acknowledges the predictor "can be turned into an agent by asking about what an agent would do — but that is under our control." Under whose control? The 6-Pack's Engagement Contracts specify what the AI may be queried for — precisely governing the scaffold Bengio leaves unspecified.
2. The aggregation problem. Non-agentic predictions, aggregated and acted upon at scale, can produce emergent agency. A recommendation algorithm predicting "this user will click" is not agentic in Bengio's sense, but at scale it shapes beliefs and behaviour. The 6-Pack's "expression ≠ amplification" principle targets exactly this.
3. Political economy. Who funds the lab? Who controls the truthification pipeline? Who chooses the flat prior? These are governance questions. Bengio's framework assumes benevolent operators. The 6-Pack assumes operators need constraining.
Yes — it is designed to. The governance architecture functions as an immune system that does not need to predict the specific failure mode. Resource caps limit blast radius regardless of internal properties. Graduated release catches emergent agency before it scales. Community evals detect anomalous behaviour that lab red teams miss. Exit rights let communities leave dangerous systems. The anti-Singleton architecture (many bounded kamis) means no single failure cascades everywhere.
But it works better with Bengio's contributions. A non-agentic predictor doesn't try to exploit loopholes in the engagement contract. The governance layer can focus on constraining human misuse rather than fighting a two-front war against both human and machine agency.
Bengio misses collective agency. His framework addresses whether a single predictor is agentic. But non-agentic components interacting — trading, recommending, moderating, surveilling — can produce emergent goal-directed behaviour. The 6-Pack's polycentric governance is designed for this multi-agent world.
The 6-Pack misses agency measurement. Bengio's quantification (bits of description for goal-achievement) provides something the 6-Pack lacks: a measurable dimension along which to evaluate danger. A kami with low measured agency is categorically safer than one with high agency, regardless of governance constraints.
Bengio misses human agency amplification. A perfect predictor of social dynamics used to manipulate elections is non-agentic — but the human wielding it is amplified. The 6-Pack (bridging algorithms, federated T&S) directly addresses this.
Bengio's trust is inside-out: mathematical properties of the predictor radiate outward. The truthification pipeline separates known truths from uncertain claims. The Bayesian posterior provides calibrated probabilities. Epistemic correctness guarantees that high-confidence claims are not lies (asymptotically). You trust the system because you can prove properties of its architecture.
The 6-Pack's trust is outside-in: governance structures constrain from above. Trust is decomposed across six relational dimensions:
The directionality matters. Bengio answers: "Can you prove this system won't deceive?" The 6-Pack answers: "When this system fails — and it will — can the affected community detect, contest, and repair the failure?" A system with perfect epistemic correctness but no appeals process would satisfy Bengio but fail four of six 6-Pack dimensions.
Pack 3 (Competence) is the most natural integration point. Its mechanisms — decision traces showing which rule fired, which sources were consulted, what the uncertainty score was — are governance infrastructure around technical performance. Bengio's epistemic correctness could serve as the engine inside this governance shell.
If a kami's underlying model trained with Bengio's methods, the decision trace wouldn't just log 87% confidence — it would guarantee (asymptotically) that 87% means what it says. Trace completeness gains teeth when uncertainty scores are epistemically correct, not just numerically produced.
The fit is not seamless. Bengio acknowledges an asymmetry: when the system says "unknown," you cannot tell whether it genuinely lacks knowledge or is withholding. This is exactly the gap that Pack 4's community-authored evaluations would stress-test — communities submitting cases where the system says "unknown" to probe whether its ignorance is genuine or strategic.
There is also a scope mismatch. Epistemic correctness addresses propositional claims. Much of what the 6-Pack governs is not propositional: should this flood relief claim be prioritised? Does this recommendation increase relational health? Bengio handles the factual substrate; the governance architecture handles the evaluative layer on top.
Yes, but with a higher burden on its institutional machinery. Without mathematical guarantees on honesty, every claim must be verified through institutional process rather than trusted on first emission. The cost is speed and labour — appeals, eval registries, oversight boards are all human-in-the-loop mechanisms. Bengio's guarantees would reduce the load on these mechanisms, not replace them.
Bengio's weaknesses:
6-Pack's weaknesses:
Bengio's Scientist AI answers: What is true? It does not answer:
The 6-Pack answers all of these, with specific mechanisms:
| Bengio's gap | 6-Pack's mechanism |
|---|---|
| What to do about predictions? | Constitutional slow lane sets red lines; operational fast lane acts within bounds |
| Who asks the questions? | Communities, via Engagement Contracts |
| What counts as harm? | "People closest to harm define harm. They author evaluations." |
| Who resolves factual disagreement? | Bridging algorithms; epistemic line separates checkable facts from value disputes |
| What happens when the AI is wrong? | Appeals, repair logs, escrow auto-payouts, incident run-books |
| Who owns the infrastructure? | Local hardware + community ownership; kami with Civic Care Licences |
Unmistakably, yes. Three moments reveal the assumption:
The deferral: "That should be a social choice, hopefully in a democracy." This presupposes democratic institutions capable of making binding choices about AI deployment. Where do they come from?
The guardrail: "We can use it as a guardrail for unsafe, untrusted agents." Who decides what counts as unsafe? Who sets the threshold? These are governance questions, not empirical ones.
The adversarial risk: "We need to design those agents as well to make sure they don't exploit adversarial attacks on the guardrail." Who audits the guardrail? Who bears responsibility when attacks succeed?
Bengio frames the Scientist AI as value-neutral — modelling the world without prescribing action. The 6-Pack's position: value-neutrality is a coherent aspiration but an impossible achievement in practice.
The 6-Pack would press on several points:
The 6-Pack would not dismiss Bengio's aspiration. It would say: pursue value-neutrality as a regulative ideal, but subject the Scientist AI to the same governance as any other system — Engagement Contracts, community-authored evaluations, and the right of affected communities to challenge its outputs.
On paper, the division is clean: Bengio handles IS (what is true), the 6-Pack handles OUGHT (what we should do). Like Hume's original distinction.
In practice, it leaks in both directions:
IS leaks into OUGHT. A Scientist AI that says "this policy will cause X deaths" has not prescribed action, but it has dramatically shaped the deliberative landscape. The selection of which consequences to model and which scenarios to present as salient are value-laden acts within the IS domain.
OUGHT leaks into IS. The 6-Pack's governance decisions determine what questions the Scientist AI investigates. If an Alignment Assembly prioritises environmental justice, the Scientist AI models environmental impacts. The ought-framework upstream determines the scope of the is-inquiry downstream.
Rather than seeing this as a problem, the interaction is generative. The 6-Pack's deliberative processes identify what needs to be known (ought drives is). The Scientist AI provides reliable knowledge about consequences (is informs ought). Responsiveness loops check whether the knowledge was adequate (ought evaluates is). A feedback cycle, not a clean partition.
Bengio: democracy is the right venue for value choices — instrumental. The 6-Pack: "a system that optimizes outcomes while removing standing has failed the basic test of alignment" — constitutive. A perfectly trustworthy oracle deployed without participation is still misaligned, because standing is non-negotiable.
| Bengio misses | 6-Pack misses |
|---|---|
| Who controls the scaffold | Mathematical guarantees for honesty |
| Emergent agency from system interactions | Scale-independence |
| Human agency amplification through AI | Internal model properties |
| Data provenance governance | Formal adversarial robustness |
| What happens before asymptotic guarantees hold | Quantifiable agency measurement |
| Political economy of the pipeline | Epistemic foundations for eval quality |
The most natural composition: Bengio's Scientist AI as the epistemic engine inside the 6-Pack's kami architecture.
Are these the same problem at different layers, or fundamentally different problems?
Different problems that must be solved together.
Inside-the-machine trust reduces adversarial pressure on governance. If the AI doesn't lie and doesn't scheme, you only need to govern human misuse. But a perfectly honest oracle deployed by a dictatorship is still a tool of oppression.
Around-the-machine trust ensures the AI serves those it affects. But governing an opaque, potentially deceptive system is an endless arms race.
The is-ought boundary leaks productively in both directions: ought drives what needs to be known; is informs what commitments demand; responsiveness checks whether knowledge was adequate.
Bengio builds the floor. The 6-Pack builds everything above it. Neither is habitable alone.
The strongest version of either framework is the one that includes the other: epistemically correct predictions nested inside governance machinery — with affected communities holding standing to contest both what the AI says and what is done with what it says.
A non-agentic predictor of reality, bounded to serve one community's needs, governed by that community's deliberative processes, with measurable accountability at every layer.
The Scientist Kami.