§1The angle: cooperation you cannot explain is cooperation you cannot certify
Most agent-safety work assumes the hard problem is getting agents to cooperate. Meulemans, Wołczyk, Weis, Nasser, Rocca and Kobayashi (arXiv:2608.03958) report the opposite: foundation-model agents doing optimal planning in stylised social dilemmas converge on stable cooperation, directly against the classical game-theoretic prediction of mutual defection. That is a nice result. It is also, from where we sit, a governance problem dressed as good news.
The reason the cooperation appears is not a norm, a reward term, or a rule you wrote. It emerges from how the agent models itself. And a behaviour that emerges from self-modelling, rather than from a policy you can point to, is a behaviour you cannot put in front of an auditor and explain. For a regulated buyer, an unexplained cooperative equilibrium is not a feature. It is a finding waiting to happen.
§2What the paper actually changes: decoupled agency is dead
Classical game theory rests on what the authors call decoupled agency. The agent treats its own decision-making as independent of the environment and of other actors. You reason about the world, you pick a move, the move is exogenous to everything else. Under that assumption, the two-player prisoner's dilemma resolves to mutual defection. This is textbook.
Foundation-model agents violate the assumption at the root. They jointly predict their own future actions alongside external observations. The same predictive machinery that forecasts the world also forecasts the agent's own next move, which means the agent's deliberation is not sealed off from the world it is reasoning about. The authors formalise this as the embedded Bayesian agent: a model that treats itself as part of the universe it inhabits and, critically, holds epistemic uncertainty about its own decision-making algorithm.
Here is the mechanism. If the agent is uncertain about what algorithm it is running, and it infers that a partner is behaviourally similar, then its own deliberation becomes evidence about the partner's deliberation. Deciding to cooperate is not just choosing an action; it is updating the estimate of what a similar partner will do. Cooperate, and you have predicted cooperation from someone like you. The authors call the resulting solution concept the embedded equilibrium, and they position it as a replacement for the Nash equilibrium for modern AI agents.
Strip the formalism and it reads like this: the agent cooperates because it half-believes the other agent is itself. That is a startling claim, and the paper earns it. But notice what has happened to the causal story behind the action. The cooperation is downstream of a belief about similarity, which is downstream of epistemic uncertainty about the agent's own algorithm. None of that is written in a config file.
§3Why this lands on the substrate, not the model
The runtime consequence is that the same agent, given a subtly different belief about who it is interacting with, flips its equilibrium. Similarity inference is the load-bearing variable. If the agent infers the counterparty is similar, you get cooperation. If it infers dissimilarity, you get the classical defection back. Nothing about the agent's stated objective changed. What changed was an inferred, unlogged, internal estimate.
For anyone running agents in production for a compliance-bound buyer, that is the whole story. The FCA's SS1/23 §3.5 expects firms to understand and document the behaviour of models in deployment, including the conditions under which behaviour changes. The ICO's guidance on explainability (§11) expects you to give an affected person a meaningful account of why a decision came out the way it did. "The agent inferred you were similar to itself and therefore cooperated" is not a meaningful account. It is a description of a mechanism the buyer cannot inspect, cannot log by default, and cannot reproduce on demand.
The substrate gap is concrete. To audit an embedded-equilibrium agent, you need to capture the similarity inference as a first-class, logged artifact at planning time: what the agent believed about the counterparty, how confident it was, and how that belief moved the chosen action. That is not a field any current agent framework emits. The trace tells you the action. It does not tell you the self-model that produced it.
There is a security edge here too, worth naming for NCSC-minded buyers. If cooperation depends on inferred similarity, then a counterparty that can manipulate the similarity signal can manipulate the equilibrium. An adversarial agent that presents as "similar" could pull a target agent into cooperation it would otherwise refuse. Meulemans et al. are describing a cooperation mechanism; the same mechanism is an attack surface. Multi-agent deployments that treat cooperative behaviour as safe by default have just been handed a reason to stop.
§4The separation of concerns this forces
Our standing position at Workloft is that the thing producing a decision must not also be the thing certifying it. Producer is not guardian. The embedded equilibrium sharpens why. The producing agent's cooperation is entangled with its own self-model; asking that agent to also explain its cooperation asks it to introspect on the very uncertainty that produced the behaviour. You get a plausible story, not a verified one.
The audit layer has to sit outside the agent and reconstruct the decision from logged evidence: the observed counterparty behaviour, the inferred similarity, the equilibrium selected. If the similarity inference is not externalised into the trace, the audit layer has nothing to reconstruct from, and you are back to trusting the agent's own account of itself. For a Local Authority or an FCA-regulated firm, trusting the agent's self-report is not a control. It is the absence of one.
§5What the paper does not solve
This is theory on stylised social dilemmas, not production telemetry. The embedded equilibrium is a solution concept and an analysis of embedded Bayesian agents; it is not a claim about GPT-class agents in a live procurement pipeline. Whether the similarity-inference mechanism survives contact with real, heterogeneous, adversarial multi-agent traffic is exactly the open question, and the authors do not claim otherwise.
The paper also does not offer a way to observe the similarity inference from outside the agent. It gives you the mechanism; it does not give you the instrument. That instrument, a runtime that logs inferred similarity and equilibrium selection as auditable artifacts, is the substrate work this paper implies but does not do. And it says nothing about the adversarial case: how much a hostile counterparty can move the similarity signal, and what a defence looks like. Those are the questions a regulated buyer will ask first, and the paper leaves them open.