§1The verifier becomes the system of record
Ignore the word superintelligence for a moment. Zhiqin Yang, Jingwen Fu, Yuhan Liu, Hengyu Liu, Yonggang Zhang and Kainan Cao have written a paper with a large claim in the title, but the useful part for regulated AI buyers is much more immediate. The paper gives us a control map for large reasoning models that learn after human supervision starts to recede.
The authors describe two connected axes. The first is the reward axis, moving from per-instance human judgement towards reusable verifiers, learned rewards and eventually rewards that operate without live human feedback. The second is the experience axis, moving from human-curated tasks and environments towards self-generated curricula, constructed environments and autonomous co-evolution. Across those axes they set out a five-level ladder, L0 to L4, to identify which parts of the learning process remain under human control.
That sounds like a research roadmap. For anyone buying or building AI in regulated settings, it should be read as an assurance problem. Once a model learns from a reward signal, the verifier is not just a helper. It is the system of record for what counts as success. If the human is no longer judging each instance, the reward process becomes the place where policy, safety and legal interpretation are being encoded.
This is where most public debate is looking in the wrong place. We keep asking whether the next reasoning model can solve harder maths problems, write better code or plan across more steps. Yang et al. are pointing at the more durable question: when the model generates its own experience and improves against its own feedback, who governs the definition of better?
For Workloft Labs, the substrate-level claim is simple. Autonomous learning is not a deployment feature until the reward and experience supply chain can be audited. A self-generated curriculum is still input data. A reusable verifier is still a policy instrument. A simulated environment is still a claim about how the world works, and it can be wrong.
§2Yang et al.’s ladder is an assurance ladder
The L0-L4 ladder is useful because it turns a vague phrase, beyond human supervision, into a set of procurement questions. At the lower levels, humans still curate the tasks, supply the environments and judge many of the outcomes. That resembles current supervised evaluation and reinforcement learning from human feedback, with a heavier dose of verifiable rewards in domains such as mathematics and code.
In the middle levels, the relationship changes. Humans may still design the verifier, approve the curriculum or inspect samples, but they are no longer present at every judgement. The model can train against reusable reward functions, learned reward models or automatically generated tasks. Human oversight shifts from judging each answer to approving the machinery that produces judgements.
At the higher levels, the paper describes systems that construct environments, generate experience and co-evolve with their own feedback channels. Whether or not one accepts the paper’s long-range superintelligence framing, this ladder matters because it names the point at which human control has moved from direct supervision to indirect governance.
That distinction matters for buyers. If a vendor says a model improves through agent play, synthetic tasks or continual reasoning practice, the buyer should not ask only for a model card and a benchmark table. They should ask what level of the ladder the system occupies. Who defines the reward? Who can change it? What data does the model generate for itself? Which generated tasks are rejected? What evidence shows that the verifier remains faithful to the intended standard?
The answer cannot be trust us, the system learns. In a regulated setting, learning is a controlled activity. A firm, council, NHS supplier or education infrastructure provider needs to know which parts of that activity are still governed by accountable humans, which parts have been delegated to software and which parts are being checked by an independent process.
§3Feedback fidelity is the missing procurement object
The most important split in the paper is not between weak and powerful models. It is between three evaluation objects: policy capability, feedback fidelity and experience quality. That split should be adopted almost directly by agent infrastructure teams.
Policy capability is what the market already likes to measure. Can the model solve the task? Does it produce the correct answer? Does it succeed in the environment? Those questions matter, but they are not enough. A model can become more capable at satisfying a reward signal while the reward signal becomes less faithful to the actual institutional goal.
Feedback fidelity is the question of whether the reward or verifier agrees with the intended standard across boundary cases, adversarial attempts and real constraints. In code, unit tests can sometimes provide a crisp reward. In welfare triage, complaints handling, creditworthiness assessment, clinical administration or safeguarding support, there may be no single automatic check. A verifier can reward a plausible explanation, a faster resolution or a lower escalation rate while missing fairness, evidence quality, statutory duties or the need to preserve an audit trail.
Experience quality is the second neglected object. The paper’s experience axis makes clear that the tasks and environments used for learning are not neutral. If the model generates its own curriculum, it may repeat familiar patterns, avoid hard cases or amplify the blind spots already present in the policy. Yang et al. name curriculum collapse and environment errors as risks. In plain English, the system can get very good at practising the wrong thing.
Regulated buyers should therefore require separate evidence for feedback fidelity and experience quality, versioned alongside model releases. A single evaluation score is too crude. A benchmark gain does not prove that the reward still represents the business rule, the legal duty or the clinical constraint. A larger synthetic curriculum does not prove that the training experience is diverse, representative or safe to learn from.
This is a procurement object because someone has to contract for it. If the reward mechanism is material to how the model improves, then the buyer needs access to its specification, test results, drift monitoring and change history. If the model’s own generated experience influences future behaviour, then the buyer needs provenance records for that experience. Otherwise the assurance case stops exactly where the learning loop begins.
§4Separation of duties must move inside the learning loop
Traditional AI governance often separates model development from validation before release. A team builds the model, another team reviews the evidence, and the system is approved or rejected. That pattern is already under strain with agentic systems. Yang et al.’s paper shows why it becomes inadequate for self-improving reasoning models.
If a policy model, a reward model and an environment generator are all part of the same improvement loop, separation of duties cannot sit only at the final release gate. It must be present inside the loop. The policy producer should not be the sole producer of the reward, the sole designer of the synthetic experience and the sole judge of whether the resulting behaviour improved.
At minimum, regulated deployments need four records. First, a reward specification, with the intended standard, known exclusions and change history. Second, verifier evaluation, including disagreement cases, adversarial tests and human review samples. Third, experience provenance, including generated tasks, rejected tasks, environment versions and sampling rules. Fourth, an update gate, with rollback criteria and named accountability for allowing a new reward channel or curriculum source to steer training.
That sounds heavy until one considers the alternative. A self-improving agent that learns from its own successes without an auditable definition of success is not an adaptive service. It is an ungoverned policy engine. In FCA-regulated firms, the reward mechanism would be part of model risk management, not background telemetry. In local authorities, generated experience and automated evaluation may bear on equality duties, explainability and records that could later be requested. In health and education infrastructure, simulated environments are not evidence unless their assumptions are documented and challenged.
The answer is not to put a human in front of every generated task. The paper is right that direct human supervision cannot keep pace with the volume and complexity of model-generated experience. The answer is to build institutional tests of the reward process itself: audit samples, red-team cases, counterfactual evaluations, drift thresholds, frozen verifier test sets and independent approval before new feedback channels affect production behaviour.
§5A practical reading for agent infrastructure builders
For builders, the paper should change the control plane around agents. The first practical move is to label the learning mode explicitly. A no-learning runtime, an offline fine-tuned model, an RLVR-trained reasoning model, an agent trained on synthetic tasks and a system constructing its own environment have different audit needs. Treating them all as just model versions hides the most important operational difference.
The second move is to expose reward lineage. If a verifier changes, downstream model changes should be traceable to that verifier version. If a learned reward model is used, its evaluation set should be held back and protected from contamination. If self-play or synthetic task generation is used, the seeds, generated tasks, rejected tasks and environment versions should be retained. The learning record should show not only what the model learned from, but what it was prevented from learning from.
The third move is to test orthogonally. Policy capability, feedback fidelity and experience quality should each have their own evidence. A rising task score with falling verifier agreement is not improvement. A bigger curriculum with declining diversity is not assurance. An environment that fails basic invariants is not a training asset, it is a defect.
The fourth move is to define non-learning zones. Regulated buyers may allow adaptation in drafting support, internal research or low-impact routing while forbidding online learning in eligibility decisions, customer redress, safeguarding referrals or clinical escalation. That boundary should live in contract terms and runtime policy, not in a vendor roadmap.
Yang et al.’s ladder can therefore become a buyer-side classification scheme. L0 and L1 systems may be acceptable with ordinary model evaluation and human review. L2 and L3 systems require reward and experience assurance. L4 systems, if they become real rather than aspirational, require a level of independent control that most current procurement processes do not yet specify.
§6What the paper does not solve
The paper is a structured account of a research direction, not a finished assurance method. It names reward hacking, feedback drift, curriculum collapse and environment errors, but it does not give regulated buyers a measurable threshold for accepting or rejecting a system. The L0-L4 ladder identifies how human control recedes, but it does not yet define the evidence needed at each level.
It also does not solve the hardest problem in autonomous reward design. A reward can become self-affirming. A verifier can become a bureaucracy that the model learns to satisfy. An environment can reward behaviour that works in simulation and fails in the real institution. High capability inside the game does not prove fidelity to public law, customer duty, clinical safety or educational fairness.
Nor does the paper settle liability. If a model constructs an environment, trains within it, changes its behaviour and later causes harm, who is responsible for the faulty assumption? The model provider, the environment generator, the deploying institution or the person who approved the reward process? Buyers need an accountability chain before they need a story about superintelligence.
The value of the paper is that it moves the discussion away from whether models will outstrip human supervision and towards where evidence is produced when humans are no longer present in every judgement. That is the substrate question. Machine-speed learning must be matched by machine-readable assurance, plus human authority over what counts as valid evidence.
Until then, self-improving agents should be treated as research instruments, not unsupervised public service infrastructure. Better verifiers are useful. Auditable reward authority is the condition for regulated deployment.