Recurrent-Architecture on Feng's Blog

From 'A Severe Misalignment' to a Mathematics Industrial Stack: A Critical Reading and a Blueprint

Sun, 13 Sep 2026 00:00:00 +0000

Why this exists: On 2026/9/11, 25 Fields medalists published “A Severe Misalignment of AI in Mathematics.” Two days later I read the declaration once straight through and once in reverse, then translated the criticism into design: if treating proof-solving as a benchmark really is the problem, what kind of institution would fix it? I care about the next executable move, not about taking sides.

The thesis: the declaration diagnosed a real bottleneck, the community’s digestion bandwidth, but filed it under a borrowed label; the right response is not to resist industrialization but to industrialize the guild’s four functions, production, verification, transmission, and credit, one by one.

Scope: this covers the declaration text and the event timeline, the strongest opposing case, the calibration between guilds and industrialization, and the design of a four-layer industrial stack for mathematics. It does not cover the signatories’ individual positions or independent verification of OpenAI’s technical results. Every design in this article is never untested, and is just a result of brainstorming.

Prerequisites: this assumes familiarity with the norms of pure mathematics research, basic concepts of Lean / mathlib and formal verification, and the general shape of research funding and incentive structures.

The event: a declaration and the week it detonated

The declaration itself is short on paper. “A Severe Misalignment of AI in Mathematics,” signed by 25 Fields medalists running from Deligne (1978) to Deng Yu (2026), with Tao, Scholze, and Villani in between. It appeared on September 11 on Tao’s blog and at mathandai.org, organized as invitation-based co-signing in the manner of the Leiden Declaration from June.

What triggered it was a chain of events in late August and early September. The direct fuse: OpenAI announced that an internal model, running roughly 10,000 agents for 88 hours, had “solved” a Navier-Stokes blowup-class problem. Four days earlier, Buckmaster and Alpöge (Anthropic) had released a 245-page draft on the forced Euler equation and accused OpenAI of scooping it. As the authorship dispute escalated, OpenAI withdrew its sponsorship of the Caltech math marathon. The letter’s core claims: AI companies treat “solving famous problems” as a benchmark, badly misaligned with the mathematics community’s goals; rushed announcements, no formal write-ups, and recurring authorship and plagiarism problems; and without the human community digesting them, AI’s ideas “can never come alive.”

I pinned the key nodes to a timeline first, because these details have been rewritten through several rounds of retelling:

Date	Event	Reliability
2026-05	OpenAI’s internal model produces a counterexample to the Erdős unit distance conjecture; nine mathematicians (including Tsimerman) verify it	secondhand
06-02	Leiden Declaration published (eight-month consultation, concrete recommendations)	verified
07-23	ICM Philadelphia: 2026 Fields Medals (Deng Yu, Wang Hong, Pardon, Tsimerman); Tsimerman announces he is joining OpenAI	Nature / Clay
09-07	Buckmaster & Alpöge (Anthropic) post their forced-Euler result, a 245-page draft; accuse OpenAI of scooping	multi-source, disputed
09-08	OpenAI announces the 88-hour “solution” of an NS blowup (about 10,000 agents, millions of dollars of compute)	single source, unverified
09-08 ~	“Merger plan” accusations and counter-accusations; OpenAI’s blog concedes it “cannot rule out that de-identified user data helped the model”	both sides contest
09-11	This declaration; OpenAI exits the Caltech sponsorship	multi-source

One media confusion deserves its own note. OpenAI’s result (a static fluid blowup under smooth forcing) and the Clay Millennium Problem’s original statement (no forcing, smooth initial data) are related but different problems. Even if the proof holds, it almost certainly does not qualify for the Clay prize, which requires peer-reviewed publication plus a two-year waiting period. The popular claim that “the Millennium Problem was solved” is itself a product of the marketing-style communication this controversy criticizes. That reading is my inference, not settled fact.

Pin the timeline first, then argue: the declaration and the counter-declaration are both fighting for narrative, and the timeline does not take sides.

Dissection: five claims and one concept slide

On the first pass I agreed with nearly everything: benchmarks, scooping, authorship, student training, every item sounded right. On the second pass, item by item, approval and wariness rose together. I broke the declaration’s case into five claims, with hidden premises and my assessment:

#	Claim	Hidden premise	Assessment
1	Treating problem-solving as a benchmark harms mathematics	Solving problems is only a proxy for “understanding”; once the proxy becomes the target, Goodhart bites	Mechanism plausible, but asserted, not argued
2	Mass production of true/false propositions destroys the soil	Mass production has happened or is imminent	Insufficient evidence: headline results number a handful per year; anticipatory rhetoric
3	Rushed announcements cause authorship/plagiarism problems	Misconduct traces back to benchmark competition	Causal chain plausible but unproven; and it is a conduct problem, not a problem of benchmark-making itself
4	Without mathematicians digesting them, AI’s ideas never come alive	The community is an indispensable digestion system	Deeply coherent, and simultaneously a claim of power
5	Mathematics’ problems mirror society’s problems	Process-is-product fields are isomorphic to product-is-product fields	True for pure mathematics, overgeneralized

The two loudest rows are claims 2 and 3: “mass production destroys the soil” is anticipatory rhetoric, since headline-grade results appear at most a handful of times a year; and the causal chain from benchmark competition to misconduct is plausible but unproven.

More important is a slide that sits outside the table. The title condemns “treating problem-solving as a benchmark,” but nearly every harm listed in the body comes from speed, authorship, and data provenance. Those are conduct problems, not problems with benchmark-making as such. Hilbert’s 23 problems were a benchmark set by the community itself, and nobody accused them of destroying mathematics. The real variable is who owns the benchmark and what incentives it settles: community-set, honor-settled, on a ten-year clock, versus company-set, funding-narrative-settled, on a weekly clock. Same machinery; who holds the dial and whose clock it runs on is the entire difference.

Having written that dissection, my impression on the declaration: true, rhetorically skilled, its core worry sound, its argumentative structure visibly incomplete. More war cry than argument. And it commits the same rushed-publication error it condemns, written in a week. Confidence: medium-high. The text was checked word by word; the OpenAI-side details rest on a single secondhand source.

What holds: proofs are compressed packages, the community is the decompressor

The declaration’s deepest insight fits in one sentence: a proof is a compressed package; the human community is the decompressor.

A proof being “found” is not the same as it becoming knowledge. Take the 245-page forced-Euler draft. Even if every line is right, it still has to be retold by peers as a shorter story, simplified to its core mechanism, written into teaching material, and finally compressed into one intuition in some young person’s head. Only then is it alive. Every station on that chain has a human bandwidth limit. This is what the declaration sees most clearly: AI’s production rate has begun to exceed the community’s digestion bandwidth, and the bottleneck has moved from the generation side to the absorption side.

The intuition reduces to one line of accounting:

$$\frac{dB}{dt} = p - d \quad (p > d)$$

$B$ is the undigested backlog, $p$ the number of worthwhile results produced per week, $d$ the rate at which the community digests them. Nothing needs solving here; the point is that “go faster” can never fix this problem. As long as $p$ keeps rising, the backlog grows linearly. The only lever that acts on the bottleneck is raising $d$.

Several other parts of the declaration hold up too. The worry about student training is concrete: mathematicians are trained by handing students problems from a pool that is still open, with mentors teaching through problems. If that pool is swept away, or merely believed to be swept away, the pipeline starves before the facts arrive. The signature list also carries information: Tao has demonstrated AI + Lean workflows in public for years, so his signature means the letter rises above the usual resistance to new tools. The letter also concedes that AI can accelerate real research; reading it as a Luddite manifesto misreads it. For professions where the process is the product, the closing warning (“your field is next”) generalizes.

One last observation complicates the insight. “Without willing mathematicians, AI’s ideas can never come alive”: that sentence is a lament and an ace at the same time. If it is true, mathematicians still hold a monopoly on the final settlement of meaning; what is threatened is not the discipline’s survival but its attention economy. The tone of fear sits oddly with that hidden position of power. I read it as the most charged line in the letter: a distress signal and a bargaining chip.

The insight holds best for pure mathematics. Pure mathematics has almost no direct practical output; understanding itself is the product, and a theorem that is true but understood by no one barely exists for the field. For applied mathematics and engineering the sentence fails: a working algorithm does not need to be understood by everyone first. Keep this boundary in view; it recurs below.

What does not hold: borrowed rhetoric and the silence on formalization

After reading the declaration until it went stale, I extracted five hard defects. The fifth is the strangest, because the tool it ignores is the one the most AI-literate signatory has championed for years.

Paradoxical self-undermining. To meet “urgency,” it skipped the eight-month consultation used by the Leiden Declaration and was written in a week, repeating the rushed publication it condemns. Tao himself acknowledged it was “unfortunate.” A text asking an industry to slow down did not slow down.
“Misalignment” is borrowed rhetoric. In AI safety the term has a specific meaning; what the declaration actually describes is incentive misalignment and externalities. The verb is its own; the noun is borrowed. Clever for reach, loose for precision.
Authority as argument. Twenty-five medals are a credibility strategy, not a sample of “the community.” This is a joint session of the elite pure-mathematics wing: the side with positive net benefit that joined OpenAI (Tsimerman) is not in the room. In the name of the community, a faction speaks.
No asks. Against the Leiden Declaration’s concrete recommendations, this text is pure indictment. A protest without asks leaves rule-making to the labs, again. This is the defect I care about most.
Silence on formal verification, the strangest omission. “Verified but not understood” is the precise form of the crisis, and the Lean path is exactly what Tao has championed in public, and the ready-made foundation for a standards bureau. The text is silent on it. I will not guess why; the cost is clear. It gives up its own most concrete engineering proposal. This is the largest argumentative gap in the letter.

Of the five, some criticize communication strategy and some criticize argument structure. Both kinds count, but I will not blend them: reach and rigor are different ledgers. The declaration may complete a historical task by forcing the issue into public view even if its case never fully stands.

The strongest opposition, and why it does not overturn the conclusion

In fairness, the declaration deserves the strongest opponent I can build.

The case runs: benchmark competition harms no theorem’s truth value. A true proof is a net gain to human intelligence whether it arrives in 88 hours or eight years. Authorship disputes are checkable conduct disputes; they belong in an investigation, not in a verdict on an industry’s methodology. Every tool revolution (computer algebra, numerical methods) came with the same laments, and mathematics digested every tool. What the medalists actually fear is the devaluation of “first prover” glory, which is their life’s capital. The cure for the student-training worry is reforming how students are trained, not prosecuting how results are measured. A helicopter ascent is not a climb, for the mountaineer; for the humans who only need the flag on the summit (the theorem), it is.

I agree with the opposition’s three factual points: truth values are unharmed, authorship disputes are procedural, and tool revolutions have precedent. But it cannot answer digestion bandwidth. The declaration’s real claim lives not at the truth level but at the level of how knowledge is socially produced: for pure mathematics, understanding is the product, and a true-but-ununderstood theorem barely exists. The backlog is not an honor problem; it is a problem of the discipline’s capacity to reproduce itself. So the right reading is not either/or but both layers stacked: optimistic at the truth layer, watchful at the production layer. The declaration’s error is billing both layers to “benchmarks”; the opposition’s limit is refusing to look at the second layer at all.

Two bias inventories go on the record. For the declaration: appeal to authority, framing effects, availability (one incident generalized to all companies), group reinforcement (a like-minded draft in a week). For my side: collaborating with an LLM model to analyze a declaration about models is a conflict of interest. The only workable rule is to score argument quality instead of identity. The same rule applies to the signatories: the argument is in the body, not the signature block.

Calibration: guilds and industrialization are not a zero-sum replacement

My first characterization of the declaration: a lament of the old order invoking its last rights, the classic script of a guild facing industrialization. “Only the best-adapted survives.” That framing has a hidden premise: industrialization destroys guilds.

The more reliable line from technology history runs differently: leaders define paradigms, and “leaders define paradigms” does not mean “old institutions vanish.” The medieval guilds were not destroyed by the industrial revolution; they transformed into standards bodies and qualification systems. ISO, medical licensing, and peer review are all industrialized descendants of guild quality-control functions. The real precondition for industrialization to succeed was the guild ceding production while keeping a monopoly on standards and taste.

So the right question is not how to bypass the guild but how to industrialize its four functions (production, verification, transmission, honor) one by one, until the guild retreats to a position nothing else can occupy. Most of my original judgment survives: OpenAI carries ethical defects, and it is also leading where mathematics may be going; finding the next move matters more than stopping to complain. What I revised is one structural point: this is a function transfer, not a replacement. If the declaration traded lament for institution-building, it would be the natural candidate for the standards bureau. Its closing line, that everything “depends on the decisions of the humans who control the technology,” already concedes as much.

This calibration has its own boundary: function transfer requires the guild to be willing and able to take the standards-bureau seat. The declaration’s behavior (no asks) suggests that willingness is not in place. If the community keeps refusing institution-building, the “cede production, keep standards” path does not exist either, and what remains really is zero-sum.

Blueprint: a four-layer industrial stack for mathematics

With the declaration dissected and the premise recalibrated, I inverted the question into a generative one: how should mathematics’ production, verification, transmission, and credit systems be reorganized so that AI output flows into human understanding at industrial throughput, with rigor enforced by infrastructure rather than by gatekeeping?

The TRIZ-style contradiction: AI produces proofs on a weekly clock; the community digests on a yearly clock. The design goal is not to balance these two but to dissolve the contradiction. Proofs self-verify; understanding is produced on demand; mathematicians spend time only on what machines cannot do (deciding what is worth doing, and what it means); rigor is enforced by infrastructure instead of spot-checked at gates.

Round one was generation without evaluation: 22 raw ideas across five families. Production systems (proof assembly line, mathematics OEM/ODM, journals as package registries). Credit and incentives (dual currency, fine-grained attribution graph, problem futures market). Verification and quality (a math FDA, red-team swarm, benchmark disclosure standard). Transmission and understanding (understanding factory, apprenticeship 2.0, canon curator). Wild seeds (orphan theorem adoption registry, reward questions over answers, mathematics digital twin). At convergence one regularity floated up on its own: the ideas that hit digestion bandwidth, rigor, and attribution at the same time all land on the same move, which is to turn verification and settlement from a gate into infrastructure. The four-layer architecture of the industrial stack follows:

Scored against six criteria (digestion bandwidth, rigor, attribution, student pipeline, technical feasibility, institutional adoptability), three families rose to the top:

Proposal	Digestion	Rigor	Attribution	Student pipeline	Feasible	Adoptable	Verdict
Math-FDA + red-team swarm (family C)	○	●●	○	○	● (formalization is mature)	● high (Clay’s two-year rule is a prototype)	best
Dual currency + attribution graph (family B)	●	—	●●	●	● pure institutional design	○ needs community coordination	second
Understanding factory + layered bundles (family D)	●●	○	○	●●	● demo-able now	● high (a university can run it)	second
Proof assembly line (family A)	○	●	○	○	○ needs agent orchestration to mature	○ companies more willing	middle
Problem futures market (family B)	○	—	○	○	●	○ liquidity problem	watch

(●● direct hit / ● related / ○ indirect / — irrelevant)

Three paradigm routes crystallized:

“Certification First” (Math-FDA). Translate the declaration’s anger into a standard: to claim success on a major result with AI, a team must supply a formal certificate, survive a red-team bounty window, and publish a reproduction kit before the result earns a “verified” certification; the human community then confers “understood” grades afterward. The guild’s gatekeeping function upgrades into a standards bureau’s mandatory certification. Guild transformation, not guild lament.

“Understanding Economy.” Make digestion bandwidth into an industry. Proofs are ore; understanding is the finished good. Orphan theorem adoption, understanding currency, and a profession of understanding engineers finally price the transmission labor the medalists treasure.

“Continuous Credit.” The structural root of authorship disputes is winner-take-all priority. Replace it with Git-style per-lemma attribution plus transitive credit, and scooping stops paying. Not a moral improvement, a design that makes the bad strategy unprofitable.

First tests: three pilots and one load-bearing dependency

A blueprint that never meets a test is just style. The design carries three minimal real tests, each with an explicit falsification condition. Cheap to run, quick to kill:

Pilot	Content	Falsification condition
Layered proof bundle demo (1-2 weeks)	Take a recent long AI-adjacent proof (e.g., the 245-page forced-Euler draft) and generate four layers: Lean-checkable subset, human-readable proof, intuition layer (“proof comic”), dependency map	If the understanding layer cannot be produced at reasonable cost and be understood by independent readers, the understanding-factory hypothesis dies
Dual-currency workshop	One Polymath-style collaboration with full dual-currency bookkeeping	If nobody behaves differently in response to understanding currency, the honor economy cannot scale
Red-team swarm micro-pilot	A $1,000 counterexample bounty on any published AI proof	If the red team neither finds holes nor raises trust, the swarm idea is out

Before landing the blueprint I also ran a force-field analysis on the leading combination:

Driving force	Blocking force	Neutralizing move
Lean / mathlib maturity (past critical mass)	Companies have no incentive to accept certification	Make certification a prerequisite for benchmark rankings (rankings are the companies’ real currency)
Talent in the Tsimerman mold willing to migrate	Community coordination failure (who leads?)	Reuse existing Clay / IMU institutions; do not build from scratch
Companies have PR incentives to self-constrain	Young mathematicians un-incentivized to do digestion labor	Tie understanding currency to career advancement (counted in hiring and tenure review)
Open-source software supplies complete precedent	Cultural identity: “industrialization equals vulgarization”	Reframe: the honorable history from guild to standards bureau (ISO, medical licensing, both guild legacies)

There is exactly one single point of failure, and it is load-bearing: the whole stack depends on formal verification coverage. Large parts of mathematics, geometric and intuitive arguments especially, are extremely expensive to formalize. If coverage stalls, the certification layer degrades and the stack collapses toward “credit layer standing alone.” Two leading indicators to watch: mathlib’s annual growth rate, and the formalization lag for major results. Those two numbers set the schedule for everything above. If they do not move, all of this stays on paper.

Boundary conditions

Fact layer: OpenAI-side details (88 hours, 10,000 agents, the “merger plan” texts) come from a single secondhand source and must be flagged as such when cited. The reading of “Millennium Problem solved” as marketing-style communication is inference, not settled.
Judgment layer: if an independent investigation shows Buckmaster’s accusation to be false, the declaration’s “misconduct” pillar weakens, and the motive map needs re-estimation.
Design layer: the 22 ideas and three routes have met no pilot; “Math-FDA is optimal” is a matrix score, not an empirical result.
Method layer: this article is a cleaned-up record of a 2026-09-13 conversation with an AI assistant. Using a model to analyze a declaration about models is a conflict of interest; the handling rule is stated in the body, but future me should assume the bias is present.
Evidence that would change my judgment: if within 12-18 months a major AI-assisted result arrives formally verified, fully written up, properly attributed, and digested by the community without friction, the “mass production destroys the soil” prediction is falsified. Conversely, if evidence emerges that AI scooping is causing young mathematicians to systematically abandon deep specializations, my “insufficient evidence” grade on claim 2 goes up.

Open questions

Which pilot first? I lean toward the layered proof bundle demo: cheapest (1-2 weeks), and it tests the most central assumption of the four layers, the understanding layer.
Will mathandai.org’s co-signing list grow beyond Fields medalists? If so, the declaration’s representation shifts from elite wing to community, and its argumentative weight shifts with it.
When will independent verification of OpenAI’s NS result land? It tests both the declaration’s “misconduct” pillar and the credibility baseline for AI proofs.
Will the Leiden Declaration’s concrete recommendations absorb this anger into a combined text with actual asks? From indictment to institution-building is one step.
Is there an acceptable threshold for formalization coverage? If geometric and intuitive arguments can only ever be partially formalized, should the stack design a bypass for the formalization-resistant zone?

[[Q]] Eighteen months from now: did the layered proof bundle demo get run? How far did mathlib’s growth rate and the formalization lag move? Do I still agree with today’s “guild to standards bureau” judgment?

References

“A Severe Misalignment of AI in Mathematics” (declaration), Terry Tao’s blog, 2026-09-11. https://terrytao.wordpress.com/2026/09/11/a-severe-misalignment-of-ai-in-mathematics/
Declaration site (co-signing), https://mathandai.org/
Leiden Declaration, https://leidendeclaration.ai (DOI: 10.5281/zenodo.20302944, 2026-06-02)
TechCrunch, “OpenAI’s feud with mathematicians is only escalating,” 2026-09-11. https://techcrunch.com/2026/09/11/openais-feud-with-mathematicians-is-only-escalating
The Economist, “Top mathematicians are outraged by OpenAI’s methods,” 2026-09-11.
36kr Chinese report (event timeline), https://eu.36kr.com/en/p/3979724367985411
Nature, “2026 Fields Medals,” https://www.nature.com/articles/d41586-026-02169-1

The checkpoint is not the policy: a merged reading of PDM and RLT

Sun, 13 Sep 2026 00:00:00 +0000

Why this exists: In September 2026 I ingested two project reports from the same author line into my vault, a diagnostics report on Prefill–Decode Kernel Mismatch (“PDM”) and an architecture report on a Recurrent Looped Transformer (“RLT”), and cross-linked them. Reading the second right after the first, the two stopped looking like separate papers. One says the mismatch between rollout and training computation is measurable and must be contractualized; the other says it can be designed out. I drafted a merged reading in Chinese for the ingest session; this is the English archive version, reorganized around three questions: what problem each report claims, what it does about it and what that delivers, and what survives a critical read.

The thesis: PDM turns RL execution fidelity into something you can measure, record, and audit, and RLT turns it into something you can design, but neither closes the causal loop from mismatch to end-to-end training outcomes.

Scope: This covers the two reports as of September 2026 (PDM August 6, revised August 24; RLT September), their problem statements, mechanisms, evidence, and weaknesses. It does not re-derive their math, does not independently reproduce any number, and treats both as non-peer-reviewed project reports. The cross-architecture hotspot hypothesis in the critique is mine, not theirs.

Prerequisites: This assumes familiarity with autoregressive decoding and KV caches, on-policy RL basics (importance ratios, off-policy correction, support conditions), and a rough picture of state-space and linear-attention models.

The problem PDM names: the checkpoint is not the policy

The assumption I carried for years is that a checkpoint defines a policy: load the weights, and every runtime that reads them computes the same distribution. PDM opens by breaking that assumption into pieces. RL pipelines generate rollouts with a sampler (parallel prefill builds a KV cache; decode steps extend it one token at a time, under quantization, custom kernels, and a sampling transform) and score the same tokens with a trainer (usually one teacher-forced parallel pass, another operator, another reduction order, sometimes another precision). Two engines with identical parameters end up defining two token distributions.

Concretely, the report writes the sampled-token log ratio as a sum of two terms:

$$\log \rho_t = \underbrace{\log \pi^{\mathrm{trainer}}{\theta_n}(a_t \mid h_t) - \log \pi^{\mathrm{beh}}{\theta_n}(a_t \mid h_t)}{\Delta{\mathrm{kernel}}} + \underbrace{\log \pi^{\mathrm{beh}}{\theta_n}(a_t \mid h_t) - \log \pi^{\mathrm{beh}}{\theta_v}(a_t \mid h_t)}{\Delta{\mathrm{stale}}}.$$

The second term is the familiar staleness of asynchronous RL: it vanishes when the sampler and trainer are in sync. The first term does not. A fully synchronous pipeline can still be off-policy for the trainer’s own target. That sentence changed how I read every “we use a fast inference engine for rollouts” line.

The emblematic bug deserves its own paragraph, because it is three lines away from any real repo. If you sample with the sampler operator and later reconstruct the “old” probability with the trainer operator, then at zero lag your recomputed ratio is exactly 1 by construction, while the correct ratio can still differ from 1. The recomputation does not fix a drift. It hides the gap, which is the report’s phrase, and it does so structurally rather than by accident.

Then there are the cases where the gap is not a rounding detail. For softmax attention, the algebra of a KV-cached recurrence and a parallel causal pass is equivalent in exact arithmetic, but that identity is not a certificate: tiling, fusion, accumulation order, cache layout, batch shape, parallel communication, precision, quantization, and the post-logit transform all live between the two computations. For linear attention and state-space models the situation is worse: tokenwise, scan, and chunkwise implementations are different schedules, chunk size C runs from 1 (tokenwise) to L (one chunk), and changing the schedule changes floating-point parenthesization. One boundary error propagates into every later state. DeltaNet makes it personal: its state update is a data-dependent corrective write, so an error in the previous state participates in the next write rather than merely being observed downstream. Recurrent sampling against a parallelized delta-rule kernel is two policies until proven otherwise.

What I find sharpest is the contract split. Distributional parity (same forward distribution on every history the sampler reaches) and score parity (the trainer differentiates the deployed policy’s score function, or an equivalent forward-and-backward implementation) are logically distinct. Equal forward distributions at one parameter value do not imply equal Jacobians. And support is part of the contract: if hard top-k, top-p, or a precision underflow assigns zero behavior probability to an action the target retains, no downstream likelihood ratio is defined, and sampled log-probabilities cannot repair that after the fact. I had filed support under “implementation detail”. It is not; it is a precondition for the estimator to exist.

The problem RLT names: one model, four execution paths

RLT’s problem statement reads like the same diagnosis one level up. A model in a production lifecycle passes through pretraining, supervised fine-tuning, rollout sampling, and replay, and each stage has historically used its own computation path, its own masking, its own caching. The report quotes the PDM line explicitly: matching forward probabilities at one parameter value is not sufficient to establish matching policy gradients, and a shared architectural transition alone does not prove numerical kernel parity. So the replay contract in RLT is not an ops checklist bolted onto an architecture; the architecture is chosen so that the checklist becomes cheap to satisfy.

The second half of RLT’s problem is depth. In a standard transformer, the computation depth available to a token is the layer count, full stop. RLT wants reasoning depth that scales with the sequence, not with the stack: it applies a single history-dependent transition to every observed or sampled token, and the recurrent state carries continuous intermediate computation from token to token. After t tokens, the state path has traversed t times the decoder depth in logical block evaluations.

The two halves connect. If one transition serves every token and every lifecycle stage, then the structural part of the sampler-trainer mismatch disappears by construction. There is no prompt path and response path to disagree; the serving split is just a boundary in the token history. I noticed here that RLT does not claim this makes things easier. It claims this makes things consistent, and consistency is the property its own replay contract needs.

What PDM does about it

The report separates measurement from repair, and its machinery falls into five layers. I keep the table because I will want it the next time I set up an RL training stack.

Layer	What it gives you
Concepts	The $\Delta_{\mathrm{kernel}} / \Delta_{\mathrm{stale}}$ decomposition, distributional vs score parity, the support condition
Records	Immutable sampler-emitted log-probabilities plus a per-token execution signature; a trainer may append diagnostics but never replace the stored probability
Audits	A four-part fixed-weight parity check: state, forward-policy, score, and sampling parity, run independently of any optimization
Remedies	Four alignment routes, described below
Operations	Two declared modes: recurrent-target and declared parallel-target, each with guardrails

The four remedies are where opinions will differ. The first is exposure: log the raw gap and its decomposition (on an audit subset, since separating the terms requires replaying the sampler operator at current weights). The second is to make one recurrence or chunk rule part of the model semantics and execute it everywhere, so there is nothing to reconcile; the report is careful that chunking does not create stochastic tokens in advance, so exact parallel generation still needs proposal and verification. The third is precision, with a ladder I now keep as a default checklist: FP32 baseline for recurrent states, decay products, and state-update reductions; FP64 as an audit reference rather than a training default; and no hope of recovering information by casting an already-rounded BF16 state up. The report captures the route’s status in a sentence I quote a lot: precision is a mitigation, not an on-policy certificate. The underlying error recursion is worth writing once, since it explains why precision alone cannot close the contract:

$$|e_t| \le L_t,|e_{t-1}| + \delta_{\mathrm{round}} + \delta_{\mathrm{schedule}}.$$

Here $L_t$ is a local state-sensitivity factor, and the two additive terms cover arithmetic rounding and a changed schedule (scan tree, chunk boundary, materialization rule). Higher precision shrinks $\delta_{\mathrm{round}}$. It does not shrink $\delta_{\mathrm{schedule}}$, and it does not stop amplification when products of the $L_t$ are large. That is the whole argument against treating FP32 as a proof.

The fourth remedy is the one I did not expect. Take speculative decoding’s machinery and invert its roles: the recurrent implementation is the correctness authority, a cheap operator drafts, and modified rejection sampling (accept with probability $\min(1, p_i(y_i)/q_i(y_i))$, resample from the normalized positive part of $p_i - q_i$ on rejection) recovers the recurrent target up to the declared hardware numerics. The report says plainly that this helps only when drafting is cheap, acceptance is high, and verification batches well, and that it can provide no speedup when recurrent verification remains the serial bottleneck. It also draws a line I would have blurred: the proposal acceptance ratio is not the RL importance ratio. After exact verification, the committed tokens are distributed according to the recurrent target, so the record stores the target’s log-probability as the behavior log-probability.

What PDM’s measurements actually show

Four evidence blocks and one registration. I care more about the “does not establish” column than about the headline numbers.

Block	Headline	Establishes	Does not establish
Estimator check (vocab 4, horizon 6, exhaustive)	Full-trajectory IS matches the exact target gradient to $2.33 \times 10^{-16}$; raw token-local stays $0.1551$ away in $L^2$	The estimator algebra is airtight in a toy world; local IS is a valid estimator of a declared local surrogate only	Anything about full-size models
Production-path audit (two H100 nodes, 8,192-token prompt)	p95 of the absolute log-ratio: 0.09293 (dense), 0.04999 (FP16 cache), 0.04433 (FP32 cache); no support violations	The gap is measurable and, in these configurations, small	Causality; generality; the two models are not matched runs
Mechanism test (128-route gated SSM)	FP32 passes forward and score checks at about $6 \times 10^{-7}$; BF16 fails both, with up to 4.69% top-1 and 0.670% gradient-sign disagreement	Precision is a real switch between passing and failing the checks; throughput rises from 13.0K to 136.0K tok/s across chunk sizes	That these error levels matter at scale
Full-distribution stress (16 histories)	Top-1 agrees in 100% of cases; total variation is still roughly 350x the dense value for the hybrid model; FP32 lowers TV by only 4.11%	Agreement at the argmax is not agreement in distribution	Production importance of the tail
Registered experiment	DAPO-17k-Eng; 512 trajectories per step, 200 steps, four arms, three seeds, 8xH100 per arm	Nothing yet; explicitly no results	Everything causal

The registered protocol is the part I will watch. It is a full pre-registration (data gate, models, arms, seeds, evaluation) with the honest footnote that no arm is accepted and the earlier pilot is excluded from claims. I expected a scaling paper to bury that; instead it sits near the top.

What RLT does about it, and what it costs

The mental model I keep from RLT is one transition, no reset. Every token, prompt or response, advances a single state $H_t = (s_t, C_t^D)$: a recurrent output plus layerwise sliding-window caches. The end of the prompt is a serving boundary in the bookkeeping, not a change in the conditional model. Gradients follow the same path: the reference training computation differentiates through the full history (full backpropagation through time with checkpointing), and the report is precise about where approximations leak in (detaching only the recurrent output still leaves gradient paths through the KV cache and encoder memory; SWA eviction is not a stop-gradient).

Three properties matter for anyone trying to falsify or adopt this. First, serving-split invariance: under identical kernels and precision, where you split prompt from response does not change the conditional distribution. The report itself flags this as mathematical equivalence only, so numerical discrepancy is exactly the door through which PDM’s problem can walk back in. Second, the state depends only on the prefix, and the appendix warns that gradient products through $\partial s_t / \partial s_{t-1}$ alone miss paths through the cache; normalization does not bound products of the transition Jacobians. Third, cache exactness: a cached prefix can replace replay only when it matches the current parameters, positions, window semantics, and execution settings; a cache built under older weights satisfies neither the current-policy value nor the gradient requirement.

The cost side is where the report is unusually forthright. The decoder path is sequential through all T transitions, so encoder parallelism does not make prefill fully parallel, and the report states it makes no claim of a reduced-prefill speedup. Cache storage grows with encoder depth plus a bounded decoder window. Training wants full BPTT. And the depth story carries its own asterisk: unbounded structural depth is not unbounded useful depth, because gates, contraction, and learned projections can suppress what long paths contribute. I find that asterisk more interesting than the headline. The design makes depth available; learning is a separate question.

So what does RLT deliver, as of September 2026? A mechanism specification. No measured reasoning quality, no measured hardware efficiency, no scaling curves. Its claim is that if the discipline is followed, the fixed-weight parallel-recurrent mismatch has no place to live, because there is only one recurrence. That is a real design-level effect and a zero-evidence empirical effect.

The seam between the two reports

Put PDM’s contract checklist next to RLT’s design, one row at a time. This is the table I check when someone tells me an architecture “solves” the mismatch.

PDM module	RLT’s response	Status
Canonical recurrent rule	One transition for all tokens, shared by pretraining, SFT, rollout, replay	Adopted at design level
Score parity	Explicitly inherited, citing PDM	Adopted
Support and behavior records	Full record semantics; metadata cannot restore missing support	Consistent
Precision ladder	Only a note that mathematical equivalence is not numerical equality	Gap
Kernel-level audit	Cache exactness only covers “is this state current-policy”	Gap
Proposal plus recurrent verification	Not addressed	Gap

The convergence is in the diagnosis, and it is close to verbatim: both reports treat forward-probability parity as insufficient and score parity as a separate contract. The divergence is in coverage. RLT borrows PDM’s logic but not PDM’s three remaining modules, which means that by PDM’s own standard, an RLT deployment is not automatically exact; it is exactly as auditable as any other pipeline, with a better starting structure. Structural mismatch (prompt path against response path) is designed away. Execution mismatch (the same RLT transition running on different kernels, precisions, and schedules) stays in PDM’s jurisdiction.

I read that as complementarity rather than competition. RLT makes PDM’s recurrent-target mode implementable at scale, because “canonical recurrence” stops being an audit nightmare and becomes the model. PDM remains the reason you cannot call the result exact without measuring it. Each report is the other’s checklist.

Critical assessment

PDM: a strong evidence chain with an open causal loop

The measurement design is careful and the self-limiting is real: the report refuses to claim that gap magnitude implies capability loss, marks the cross-model numbers as descriptive, and labels the support stress as appendix-level evidence with a predeclared disposition. What I cannot yet see is the value proposition closed end-to-end. If production-path p95 log-ratios sit at 0.04 to 0.09 and even the full-distribution stress shows top-1 agreement across the board, the honest reading of the current data is that the gap is real, measurable, and modest in tested configurations. That makes this agenda insurance and auditability first, performance repair second, until the registered experiment says otherwise. I would also want a cost ledger: recurrent replay gives up sequence parallelism, the precision ladder costs compute, and proposal-verification can provide no speedup at all. Nobody has priced the discipline against the gap it removes.

Two smaller critiques. First, the empirical surface is small: one 8,192-token prompt path, sixteen stress histories, and a 128-route toy SSM. These demonstrate measurability, not prevalence. Second, much of the toolbox is assembled from known parts (importance sampling, modified rejection, FP32 baselines); the new contribution is the execution-level framing and the systematic contractualization. That is a framework contribution, and expecting an algorithmic surprise from it would be a category error.

RLT: design discipline with zero measurements

The report is unusually honest for a design proposal, and that honesty is also its evaluation. It states no measured results, concedes that structural depth is not a reasoning guarantee, and concedes that neither weight tying nor temporal recurrence alone establishes novelty. What remains is a system-consistency argument: a lifecycle with one transition, replay semantics that follow from the architecture, and explicit interfaces for caches, multi-turn serving, and gradient approximations. The costs point the other way from its selling point: prefill loses parallelism, memory grows, and training wants full BPTT. Whether a quality-per-cost sweet spot exists is exactly the experiment that does not exist yet. And by PDM’s standards, RLT’s exact-replay claim is itself pending audit: no precision ladder, no kernel-level audit, and no treatment of stochastic computation beyond a deterministic-forward assumption.

What both leave open together

The full narrative chain reads: diagnose the execution mismatch (PDM), design it away (RLT), show the training outcome change. The first link is measured, the second is specified, the third is empty. And there is no economics anywhere in either report. Both add cost in the name of correctness, and neither answers at what training scale the discipline pays for itself. My own working hypothesis, flagged as inference rather than result, is that the payoff is highly architecture-dependent: I expect dense softmax configurations to stay in the modest regime, while chunkwise state-space and delta-rule models carry structural gaps where alignment discipline could be the difference between a stable run and a mysterious one. A systematic gap taxonomy (architecture family against kernel pair against precision) does not exist. That matrix is the most valuable missing artifact I can see in this area.

What I take from reading them together

Two hooks to keep. PDM: the checkpoint name is not the policy; the policy is the executable distribution, and an exact deployment gradient also includes its score function. RLT: one transition for the whole lifecycle, or replay fidelity becomes someone else’s audit problem.

Three operational items, in order of how soon they apply to any RL stack I touch. One, audit for the recomputation bug: if old probabilities are rebuilt with the trainer engine, the kernel gap is being hidden, and the fix is to store sampler-emitted probabilities at sampling time. Two, adopt a minimal record contract before scaling: sampler log-probability, version, execution signature, sampling metadata, all immutable. Three, declare the target mode (recurrent-target or declared parallel-target) and do not blur it; the blur is where silent off-policyness lives.

The watch point is the registered DAPO experiment. When it runs, it becomes the first causal data on whether any of this changes training outcomes, and which arm (naive, raw local IS, stabilized IS, FP32 mitigation) moves first. My prior is mild: I expect FP32 state handling to matter more than ratio corrections for hybrid models, and neither to be dramatic on dense ones. I would like to be wrong in a specific direction. If the stabilized arm separates from naive at 200 steps, the priority of this whole line changes from insurance to intervention.

Boundary conditions

Every number in this reading is as of September 2026, and PDM is already on its second revision (August 24). Treat figures as versioned, not fixed.
The “modest gap” reading is bounded by the tested configurations: 4B-class models, one 8,192-token prompt path, sixteen stress histories. It does not extend to frontier scale or other architectures.
The hotspot hypothesis (structural architectures carry larger gaps) is mine. It could be wrong in either direction, and the registered experiment will only test two Qwen models.
RLT’s cost profile is argued, not measured. A working implementation could come out better or worse than the design reasoning suggests.
Both reports share one unexamined premise: that execution fidelity, once measurable, deserves to be optimized for. If audit magnitude turns out uncorrelated with outcomes, the agenda shifts from engineering priority to bookkeeping.
I read both as non-peer-reviewed project reports; the absence of external review is a boundary on everything above.

Open questions

When the DAPO registration runs, which arm separates from naive first: stabilized token-local IS, the FP32 recurrent-state mitigation, or none of them within 200 steps?
What does the actual delta-kernel matrix look like across architecture families at fixed scale, and is there a cheap pre-training predictor for “this kernel pair will bite”?
Can the four-part parity audit run as a CI gate inside a training stack, and what does one audit step cost relative to one optimizer step?
Does RLT’s serving-split invariance survive a production decode stack, or does hardware entropy re-open PDM’s gap the moment the sampler is not the reference kernel?
Is there a regime where full-BPTT recurrent replay is cheaper than the mismatch it removes, for example short-response verifiable-reward RL where outcome variance dominates?

[[Q]] Six months from now: has the DAPO-17k-Eng registration produced accepted results, and did any kernel-mismatch arm beat the naive baseline in evaluation? If not, check whether the blocker was infrastructure, a support violation, or a genuinely null effect.

References

Yifan Zhang et al., “Reliable RL Scaling Requires Accounting for Prefill–Decode Kernel Mismatch”, Pretraining-RL-Science project report, August 6, 2026 (revised August 24, 2026). https://github.com/yifanzhang-pro/Pretraining-RL-Science
Yifan Zhang et al., “Recurrent Looped Transformer”, project report, September 2026. https://github.com/yifanzhang-pro/recurrent-looped-tranformer

Qwen3-TTS on AMD Radeon 780M (gfx1103): A Docker + ROCm Deployment Autopsy

Mon, 03 Aug 2026 00:00:00 +0000

Qwen3-TTS on AMD Radeon 780M (gfx1103): A Docker + ROCm Deployment Autopsy

Abstract. This document is a complete post-mortem of deploying the Qwen3-TTS-Openai-Fastapi service (OpenAI-compatible TTS API, Qwen3-TTS-12Hz- 1.7B-CustomVoice model, default voice Vivian) on an AMD Phoenix1 Radeon 780M (gfx1103) APU using Docker with an ROCm container. It is written for the author’s future self and for engineers who will deploy LLM / MoE inference on the same hardware class. The central finding: the per-token .item() GPU-to- CPU synchronization inside the Hugging Face transformers generate() loop permanently deadlocks on this APU because HSA completion signals are unreliable, while short texts survive only because they trigger few syncs. Routing around generate() with the project’s built-in fast codebook path eliminates the syncs and stabilizes the service at RTF ~2.32. Every command, error message, and observation in this document comes from the actual 2026-08-02 debugging session.

1. Background: running TTS on a 780M

In early August 2026 I needed a self-hosted TTS service reachable over HTTP on the local network, able to synthesize Chinese and English, running as a persistent container whose configuration survives restarts. I chose Qwen3-TTS-Openai-Fastapi with the Qwen3-TTS-12Hz-1.7B-CustomVoice model and Vivian as the default voice.

The hardware was an AMD Phoenix1 Radeon 780M (gfx1103), an APU integrated GPU that shares system memory and has no dedicated VRAM. gfx1103 does not appear in the official ROCm supported-device list. The host passes only /dev/kfd and /dev/dri/renderD128 into the container; all ROCm userspace lives inside the image. That design decision was the root of most subsequent trouble.

Debugging ran from 10 pm into the early morning: four files modified, several image rebuilds, and three distinct forms of GPU deadlock. This article records the full chain in chronological order and closes with a distilled checklist intended as a pitfall manual for future LLM / MoE deployments.

2. Environment bring-up: image, dependencies, kernel masquerade

2.1 The base image tag never existed

The repository’s Dockerfile.rocm referenced rocm6.3.1_ubuntu22.04_py3.12_pytorch_release_2.6.0. The build failed with “not found” from Docker Hub. Checking the Hub’s tag listing confirmed that this exact combination was never published.

I switched to rocm6.4.3_ubuntu24.04_py3.12_pytorch_release_2.6.0: identical torch 2.6.0 and Python 3.12, differing only in the Ubuntu base layer. I deliberately avoided ROCm 7.x because the community reports SIGSEGV on the 780M; the 6.x line is comparatively stable.

FROM rocm/pytorch:rocm6.4.3_ubuntu24.04_py3.12_pytorch_release_2.6.0

2.2 pip silently replaced torchaudio with the CUDA build

The first container start failed with libcudart.so.13: cannot open shared object file. The cause was subtle: the project’s pyproject.toml declares a bare torchaudio dependency with no version constraint, so pip resolved the then-latest CUDA build (torchaudio 2.11.0), overwriting the image’s ROCm torch stack and pulling in CUDA-only libcudart.

The fix was a forced reinstall of the matching ROCm build after pip install -e ".[api]", from PyTorch’s official ROCm wheel index:

pip install --no-cache-dir --no-deps \
 torchaudio==2.6.0+rocm6.2.4 \
 --index-url https://download.pytorch.org/whl/rocm6.2.4

Lesson: treat every bare dependency inside a ROCm image with suspicion. pip defaults to the newest CUDA build; lock versions explicitly with a +rocm suffix.

2.3 gfx1103 has no precompiled kernels: HSA_OVERRIDE_GFX_VERSION

Model loading failed with HIP error: invalid device function because gfx1103 is absent from the ROCm 6.4.3 precompiled-kernel list (which covers gfx908/90a/1030/1100/1101/942 and others). The standard remedy is the HSA_OVERRIDE_GFX_VERSION environment variable, which masquerades the chip as a different model.

Here is the counterintuitive trap: following my ollama deployment notes I first set 11.0.2 (the real version for gfx1103) and still got “invalid device function”, because the image contains no kernels compiled for 11.0.2. The value that works is:

HSA_OVERRIDE_GFX_VERSION=11.0.0 # masquerade as gfx1100 (desktop RDNA3)

11.0.0 maps to gfx1100; the desktop RDNA3 kernels ship in the image and load directly. Set this in compose environment (it overrides the Dockerfile ENV).

3. First wave of GPU hangs: flash-attn, AOTriton, and tunableop

With the model loadable, every warmup now hit the same hard error:

HW Exception by GPU node-1 reason: GPU Hang

The GPU side deadlocked completely and the driver required a reset. I eliminated suspects one at a time.

3.1 flash-attn compiled, then proved useless

The Dockerfile compiles flash-attn v2.8.3 from source. It compiled successfully and the model even loaded in flash_attention_2 mode, but warmup always hung. After switching attention to sdpa in the config, loading worked. Conclusion: under the gfx1103 masquerade, the flash-attn kernel path is unstable; sdpa is the only viable choice.

3.2 TORCH_SDPA_ENABLE_FLASH=0 was a false positive

Logs repeatedly showed “Using AOTriton backend for Flash Attention forward”. PyTorch 2.6 on ROCm routes SDPA through an AOTriton flash implementation by default, so I tried disabling it with TORCH_SDPA_ENABLE_FLASH=0. My test script passed, which looked like success.

It was a false positive. The test used float32 inputs, and flash attention does not support float32, so it silently fell back to the efficient backend regardless of the environment variable. With bf16 inputs the variable had no effect and the flash path ran anyway. The correct way is to force-disable it in Python:

import torch
torch.backends.cuda.enable_flash_sdp(False)
torch.backends.cuda.enable_mem_efficient_sdp(True)
torch.backends.cuda.enable_math_sdp(True)

To make this apply automatically at process start, I wrote a sitecustomize.py into the image’s site-packages; Python imports it at startup, so no application code needed modification.

3.3 PYTORCH_TUNABLEOP_ENABLED=1 was the hidden bomb

This was the first fix that produced visible progress. The image ships with PYTORCH_TUNABLEOP_ENABLED=1, which runs a kernel-tuning benchmark on first call of every operator. On the APU that tuning process hangs the GPU. Setting it to 0 let warmup 3/3 complete, the service came up for the first time, and short-text synthesis worked.

At this point short text (“你好世界。” / “Hello, world.”) returned reliably in 2.6 seconds. I assumed debugging was over.

4. The main culprit: per-step .item() sync inside transformers generate()

4.1 Symptom: short text fine, medium text always hangs

The service answered short requests, but any request of moderate length (about 30 characters) hung forever. curl timed out at 120 seconds, and the container health went from healthy to starting (warmup wedged). The symptom was highly reproducible: short text 100% success, medium text 100% hang.

GPU monitoring showed an odd pattern:

t=1s busy=74% # generation in progress
t=9s busy=96% # GPU under load
t=10s busy=0 # GPU suddenly completely idle
t=11s+ busy=0 # stays idle, but the request never returns

The GPU finished its work and went idle, while two CPU threads spun at 99.9% each. This was not a GPU computation problem; the CPU was waiting for something that never arrived.

4.2 Double evidence from py-spy and gdb

After granting the container SYS_PTRACE, py-spy showed the Python stack:

_has_unfinished_sequences (transformers/generation/utils.py:2597)
_sample (transformers/generation/utils.py:2779)
generate (transformers/generation/utils.py:2564)
forward (qwen_tts/core/models/modeling_qwen3_tts.py:1891)

Wedged inside the transformers generation loop’s termination check. gdb then showed the C stack:

_local_scalar_dense_cuda -> .item() GPU-to-CPU sync copy
-> libhsa-runtime64.so.1 signal wait

The chain was complete: _has_unfinished_sequences calls tensor.item() after every generated token, copying the result from GPU to CPU to decide whether to stop. .item() is a D2H copy that depends on an HSA completion signal. On the 780M that signal is unreliable: the kernel actually finishes, but the signal is lost and the CPU blocks forever.

Short text survived because it performs few syncs and happened to receive every signal. Long text performs many syncs, and the probability of hitting a lost signal approaches one. This is a probabilistic failure, not a length threshold.

4.3 One change that made things worse: HSA_ENABLE_INTERRUPT=0

At one point I set HSA_ENABLE_INTERRUPT=0 to make HSA poll instead of using interrupts. That converted “waiting for a signal” into CPU spinning: the symptom changed from GPU Hang to two threads at 99.9% CPU, which was harder to diagnose. Looking back at the timeline, the first stable run (warmup passed, short text worked) happened under the default interrupt mode, before this variable was introduced. I removed it and went back to defaults.

5. The fix: non_streaming_mode and fast codebook

Once the .item() sync was identified, the strategy became: bypass transformers’ generate() loop so no per-token synchronization happens. The project code happens to provide two routes.

5.1 First route: non_streaming_mode=True

generate_custom_voice defaults to non_streaming_mode=False, which simulates streaming input and runs the per-token loop (a .item() on every step). Passing True runs one-shot generation and cuts the sync count by an order of magnitude. I patched the optimized backend’s call:

wavs, sr = self.model.generate_custom_voice(
 text=text, language=language, speaker=voice, instruct=instruct,
 non_streaming_mode=True,
)

An isolated test confirmed the diagnosis: for identical short text, the streaming-simulation mode deadlocked while non_streaming mode succeeded in 4.4 seconds. This proved the root cause but did not fully cure it: medium text still hung intermittently through the API.

5.2 Second route: fast codebook

The real cure is enable_streaming_optimizations(use_fast_codebook=True). It replaces the inner code_predictor’s Hugging Face generate() with the project’s own generate_fast: pure tensor operations (topk / multinomial / masked_fill, all on GPU), zero .item() syncs, and a fixed iteration count of num_codebooks with no CPU involvement in the stop decision.

In the original code the fast codebook switch was gated behind use_compile, so use_compile: false skipped the entire _apply_optimizations block. I changed the condition so either flag enables the block, and made the hardcoded use_compile=True read from config:

if (opt.get("use_compile", True) or opt.get("use_fast_codebook", True)) \
 and self.device != "cpu":
 await self._apply_optimizations(model_key, model_info, opt)

Fast codebook is now enabled independently, while torch.compile stays off (compilation is both slow and risky on the APU).

5.3 Supporting stability parameters

Final compose environment:

HSA_OVERRIDE_GFX_VERSION=11.0.0
HSA_ENABLE_SDMA=0 # known iGPU hang source
PYTORCH_TUNABLEOP_ENABLED=0
TTS_AUTOCHUNK=false # chunked continuous generation also hung; one-pass now
TTS_LAZY_LOAD=false

HSA_ENABLE_SDMA=0 is a community-known iGPU stability switch. TTS_AUTOCHUNK=false came from observing that multi-chunk continuous generation also triggered hangs; with it off, long text synthesizes in one pass.

6. Verification and performance

Benchmark after the fix (1.7B + sdpa + fast codebook, no torch.compile):

test	wall time	audio	RTF
English short	6.8s	2.9s	2.32
English medium	27.4s	11.9s	2.32
Chinese medium	20.0s	8.6s	2.32
overall p95	—	—	2.34

RTF 2.32 means 2.3 seconds of compute per second of audio. For a 780M integrated GPU that is a reasonable level: batch synthesis is fully usable; real-time conversation (RTF < 1) is not.

The long-text validation was the real relief: synthesizing the full “蜀道难” (Hard Is the Road to Shu, a 500+ character classical Chinese poem) in one pass produced 119.8 seconds of audio (24 kHz / 16-bit WAV) in 5 minutes 38 seconds with no hangs and steady GPU load. At this length the failure rate used to be 100%.

One data trap surfaced along the way: the local directory labeled “0.6B” contained a byte-for-byte copy of the 1.7B model (identical MD5, config tts_model_size: "1b7"). The real 0.6B (905M parameters) lived in another legacy directory. Never trust a directory name when switching models.

7. Checklist for future LLM / MoE deployments

Verify the base image tag exists before writing the Dockerfile. The Docker Hub tag listing is paginated and searchable; five minutes of checking saves a failed build.
Pin every torch-adjacent dependency inside ROCm images. Bare dependencies get resolved by pip to the newest CUDA build. Use +rocm<ver> suffixes and the official ROCm wheel index uniformly.
For gfx1103, set HSA_OVERRIDE_GFX_VERSION=11.0.0 (masquerade as gfx1100). 11.0.2 / 11.0.3 are the true gfx1103 version numbers but no kernels exist for them in the image, producing “invalid device function”. Set it in compose environment so it overrides the Dockerfile ENV.
Disable PYTORCH_TUNABLEOP_ENABLED on APUs. The kernel-tuning benchmark hangs the integrated GPU; it is the easiest hidden bomb to miss.
Disable flash-attn and the AOTriton flash SDPA. flash-attn compiles but hangs at runtime; for SDPA, force enable_flash_sdp(False) via a sitecustomize.py so it applies at Python startup. The TORCH_SDPA_ENABLE_FLASH=0 environment variable is unreliable on ROCm, and any test of it must use bf16 inputs or it is a false positive.
Watch for per-token .item() syncs inside transformers generate(). Any GPU-to-CPU synchronization in the generation loop is a time bomb on the 780M. Prefer sync-free generation paths (here: fast codebook), or move the termination check onto the GPU.
Leave HSA_ENABLE_INTERRUPT at its default. Setting it to 0 turns lost signals into CPU spinning, which is much harder to diagnose.
Diagnostic toolkit: py-spy for Python stacks, gdb for C stacks, and gpu_busy_percent for GPU load. Cross-referencing them quickly distinguishes “GPU cannot compute” from “CPU waits forever”.
Test across a length gradient. Short-text success does not imply long-text success; signal loss is probabilistic and scales with sync count.
Persistence trio: restart: unless-stopped, bind-mounted model and config, and bind-mounted cache directories (~/.cache/miopen, torch_extensions). Otherwise a container recreate forces kernel re-search/recompilation and can reproduce the hang.
Locking the GPU frequency to high improves stability: /sys/class/drm/card0/device/power_dpm_force_performance_level set to “high” requires root and resets on reboot; persist it with a systemd oneshot service.
Verify model authenticity: cross-check the directory name, the config’s model_size field, and file MD5s. Do not trust a copied directory.

8. Boundary conditions

The findings hold for ROCm 6.4.3 + torch 2.6.0 + Ubuntu 24.04. With other ROCm versions (especially 7.x), kernel coverage and AOTriton behavior change, and the HSA override value may need retesting.
Masquerading gfx1103 as gfx1100 is a compatibility shim; performance is whatever it measures to be. Other APUs (gfx1101/1102, Strix Point) may need different stable parameter sets.
The fast codebook path bypasses the HF generate() of this project’s code_predictor only. LLM / MoE inference that must use transformers generate() (complex sampling, stopping criteria) cannot adopt this fix directly; a sync-free path must be found separately.
RTF 2.32 is “usable”, not “good”. Streaming endpoints, voice cloning, and Base (non-CustomVoice) models were not validated in this deployment and may still trigger hangs.
Short-text success in early versions (2.6s) was luck: few syncs. It must not be taken as evidence that a configuration works; always re-test with medium or long text.

9. Open questions

What RTF does the real 0.6B model (905M parameters) achieve under the same configuration? If it approaches 1.0, does real-time become viable?
Is there a usable torch.compile configuration on the 780M (e.g. reduce-overhead plus manual warmup)? If so, how far below RTF 2.32 can we go?
Has the ROCm 7.x SIGSEGV on gfx1103 been fixed? If yes, is migrating worth it for better kernel coverage?
Is the streaming endpoint (/v1/audio/speech streaming mode) stable under fast codebook? This decides voice-agent viability.
For LLM / MoE inference (e.g. vLLM’s ROCm support), do the HSA override and tunableop lessons transfer directly, or does vLLM have its own APU parameter system?

[[Q]] Six months from now: is there a fully sync-free generation path (including a custom stopping criterion) that makes transformers generate() run stable long sequences on the 780M?

10. References

Qwen3-TTS-Openai-Fastapi repository (Dockerfile.rocm, config.yaml, api/backends/optimized_backend.py), local copy, August 2026.
Docker Hub rocm/pytorch image tag listing, queried August 2026.
Qwen3-TTS-12Hz-1.7B-CustomVoice model config.json (local model directory).
PyTorch ROCm wheel index, download.pytorch.org/whl/rocm6.2.4, accessed August 2026.
AMD ROCm community discussions on gfx1103 and HSA_OVERRIDE_GFX_VERSION (web search, August 2026).

Qwen3-TTS 在 AMD Radeon 780M 上的 Docker + ROCm 部署排障记录

Mon, 03 Aug 2026 00:00:00 +0000

Qwen3-TTS 在 AMD Radeon 780M 上的 Docker + ROCm 部署排障记录

背景：在 780M 上跑 TTS 的目标与约束

2026 年 8 月初，我要在本地部署一个自托管的 TTS 服务，选型是 Qwen3-TTS-Openai-Fastapi（OpenAI 兼容接口）加 Qwen3-TTS-12Hz-1.7B-CustomVoice 模型，默认音色 Vivian。目标很朴素：局域网内能调 HTTP API 合成中英文语音，容器持久运行，重启不丢配置。

硬件是一个我此前没在 PyTorch 场景认真用过的家伙：AMD Phoenix1 Radeon 780M (gfx1103)，APU 集成显卡。它共享系统内存，没有独立显存，官方 ROCm 支持列表里没有 gfx1103 这个名字。宿主机只透传 /dev/kfd 和 /dev/dri/renderD128 进容器， ROCm 用户态全部住在镜像里，这是后来所有麻烦的根源。

排障从晚上 10 点持续到凌晨，前后改了 4 个文件、重建镜像多次、经历了三种不同形态的 GPU 卡死。这篇文章按时间顺序记录完整链路，末尾是提炼出的经验清单，目标是给之后部署 LLM / MoE 模型时当避坑手册用。

环境搭建：镜像、依赖与内核伪装

第一步就踩空：镜像 tag 不存在

仓库自带的 Dockerfile.rocm 引用 rocm6.3.1_ubuntu22.04_py3.12_pytorch_release_2.6.0，构建时 Docker Hub 直接返回 not found。查了 Docker Hub 的 tag 列表，发现 6.3.1 这个组合从未发布过。

改用 rocm6.4.3_ubuntu24.04_py3.12_pytorch_release_2.6.0：torch 2.6.0 相同、 Python 3.12 相同，只差 Ubuntu 基础层。特意避开 ROCm 7.x，因为社区在 780M 上有 SIGSEGV 报告，6.x 系列相对稳。

FROM rocm/pytorch:rocm6.4.3_ubuntu24.04_py3.12_pytorch_release_2.6.0

torchaudio 被 pip 覆盖成 CUDA 版

第一次启动容器，日志报 libcudart.so.13: cannot open shared object file。根因很隐蔽：项目的 pyproject.toml 里裸依赖 torchaudio（没有版本约束）， pip 解析时装到了当时最新的 CUDA 版 torchaudio 2.11.0，把镜像里 ROCm 的 torch 栈覆盖掉了，还引入了 CUDA 专属的 libcudart。

修复是在 pip install -e ".[api]" 之后强制重装 ROCm 配套版本，并从 PyTorch 官方 ROCm wheel 源拉取：

pip install --no-cache-dir --no-deps \
 torchaudio==2.6.0+rocm6.2.4 \
 --index-url https://download.pytorch.org/whl/rocm6.2.4

经验：ROCm 镜像里任何裸依赖都要警惕，pip 默认解析到 CUDA 最新版，必须显式锁定 +rocm 后缀的版本。

gfx1103 没有预编译 kernel：HSA_OVERRIDE_GFX_VERSION

模型加载时报 HIP error: invalid device function。原因是 gfx1103 不在 ROCm 6.4.3 预编译 kernel 列表里（列表覆盖 gfx908/90a/1030/1100/1101/942 等）。解决办法是环境变量 HSA_OVERRIDE_GFX_VERSION 把芯片伪装成其他型号。

这里有个反直觉的坑：我最初按 ollama 部署经验设了 11.0.2（gfx1103 的真实版本号），结果反而报 invalid device function，因为镜像里根本没有为 11.0.2 编译的 kernel。最终生效值是：

HSA_OVERRIDE_GFX_VERSION=11.0.0 # 伪装成 gfx1100（桌面 RDNA3）

11.0.0 对应 gfx1100，桌面 RDNA3 kernel 在镜像里自带，加载后直接可用。这个值必须写在 compose 的 environment 里（会覆盖 Dockerfile 的 ENV）。

第一波 GPU Hang：flash-attn、AOTriton 与 tunableop

模型能加载了，但每次 warmup 都触发同一个硬错误：

HW Exception by GPU node-1 reason: GPU Hang

这条错误意味着 GPU 侧彻底死锁，驱动需要重置。我按嫌疑顺序一个个排除：

flash-attn 编译成功但没有意义

Dockerfile 里从源码编译了 flash-attn v2.8.3，居然编译成功，模型也确实以 flash_attention_2 模式加载了。但 warmup 必挂。把 config 里 attention 改成 sdpa 之后，加载正常了。结论：在 gfx1103 伪装环境下，flash-attn 的 kernel 路径不稳定，用 sdpa 是唯一选择。

TORCH_SDPA_ENABLE_FLASH=0 是假阳性

日志里反复出现 “Using AOTriton backend for Flash Attention forward”。 PyTorch 2.6 ROCm 的 SDPA 默认走 AOTriton flash 实现，我试着用环境变量 TORCH_SDPA_ENABLE_FLASH=0 关掉它。测试脚本跑通了，看起来有效。

但后来发现这是假阳性：测试用的是 float32 输入，flash attention 本来就不支持 fp32，自动降级到 efficient backend，跟环境变量无关。换成 bf16 输入后，环境变量完全失效，flash 路径照走。正确做法是在 Python 里强制关：

import torch
torch.backends.cuda.enable_flash_sdp(False)
torch.backends.cuda.enable_mem_efficient_sdp(True)
torch.backends.cuda.enable_math_sdp(True)

为了让服务进程启动时自动执行，我在 Dockerfile 里写了一个 sitecustomize.py 放进 site-packages，Python 启动时自动加载，不需要改业务代码。

PYTORCH_TUNABLEOP_ENABLED=1 是隐藏炸弹

这是第一个真正让我看到转机的修复。镜像自带的 PYTORCH_TUNABLEOP_ENABLED=1 会在首次调用每个算子时做 kernel 调优基准，在 APU 上这个调优过程会挂 GPU。设成 0 之后，warmup 3/3 全部通过，服务第一次真正起来了，短文本合成成功。

到这里，短文本（“你好世界。"）稳定 2.6 秒出结果。我一度以为排障结束了。

主犯：transformers generate() 的 .item() 同步

症状：短文本好，长文本必挂

服务能响应短文本，但任意中等长度文本（约 30 个字符）请求就会永久挂起。 curl 等 120 秒超时，容器 health 从 healthy 变 starting（warmup 挂住）。这个症状高度稳定：短文本 100% 成功，中文本 100% 挂。

监控 GPU 时发现一个反常现象：

t=1s busy=74% # 生成进行中
t=9s busy=96% # GPU 高负载
t=10s busy=0 # GPU 突然完全空闲
t=11s+ busy=0 # 一直空闲，但请求永不返回

GPU 干完活就停了，但 CPU 侧有两个线程各占 99.9%，在空转。这不是 GPU 算不动，是 CPU 在等一个永远不来的东西。

py-spy 和 gdb 的双重证据

给容器加了 SYS_PTRACE 权限后，用 py-spy 抓 Python 栈：

_has_unfinished_sequences (transformers/generation/utils.py:2597)
_sample (transformers/generation/utils.py:2779)
generate (transformers/generation/utils.py:2564)
forward (qwen_tts/core/models/modeling_qwen3_tts.py:1891)

卡在 transformers 生成循环的结束条件判断里。再用 gdb 抓 C 栈：

_local_scalar_dense_cuda -> .item() GPU→CPU 同步拷贝
-> libhsa-runtime64.so.1 信号等待

链条完整了：transformers 的 _has_unfinished_sequences 每生成一个 token 就调用 tensor.item()，把 GPU 上的结果同步回 CPU 判断是否该停。.item() 本质是 GPU→CPU 的 D2H 拷贝，依赖 HSA 信号通知完成。在 780M 上这个信号不可靠，kernel 实际跑完了，但信号丢失，CPU 永久阻塞在等待里。

短文本为什么能过？因为同步次数少，碰巧每次都拿到信号；长文本同步次数多了，撞上信号丢失的概率接近 1。这是概率性问题，不是长度阈值问题。

一个让事情更糟的尝试：HSA_ENABLE_INTERRUPT=0

我一度设置 HSA_ENABLE_INTERRUPT=0 让 HSA 用轮询代替中断，结果把 “等信号"变成了 CPU 自旋，症状从 GPU Hang 变成双线程 99.9% CPU 空转，更加难排查。回看时间线，服务第一次稳定跑通时（warmup 全过、短文本成功）恰恰是在没设置这个变量的默认中断模式下。撤掉它，恢复默认。

修复组合：non_streaming_mode 与 fast codebook

定位到 .item() 同步之后，思路变成：绕过 transformers 的 generate() 循环，不逐 token 同步。项目代码里恰好有两条路。

第一条路：non_streaming_mode=True

generate_custom_voice 默认 non_streaming_mode=False，这个模式会模拟流式输入，走逐 token 的生成循环（每步 .item()）。传 True 走一次性生成，同步次数少一个量级。我改了 optimized_backend 的调用：

wavs, sr = self.model.generate_custom_voice(
 text=text, language=language, speaker=voice, instruct=instruct,
 non_streaming_mode=True,
)

隔离测试确认：同样的短文本，流式模拟模式卡死，non_streaming 模式 4.4 秒成功。这证明根因判断正确，但还没根治：中等文本在 API 里仍偶发卡住。

第二条路：fast codebook

真正的解药是 enable_streaming_optimizations(use_fast_codebook=True)。它把内层 code_predictor 的 HF generate() 换成项目自带的 generate_fast：纯 tensor 操作（topk / multinomial / masked_fill 全在 GPU 上），零 .item() 同步，固定迭代 num_codebooks 次，不需要任何 CPU 参与判断。

原代码里 fast codebook 的开关被挂在 use_compile 条件后面：use_compile: false 时整个 _apply_optimizations 被跳过。我把条件改成两者任一为真就执行，并把硬编码的 use_compile=True 改成从 config 读取：

if (opt.get("use_compile", True) or opt.get("use_fast_codebook", True)) \
 and self.device != "cpu":
 await self._apply_optimizations(model_key, model_info, opt)

这样 fast codebook 独立启用，torch.compile 保持关闭（APU 上编译又慢又危险）。

配套稳定化参数

最终 compose 环境变量组合：

HSA_OVERRIDE_GFX_VERSION=11.0.0
HSA_ENABLE_SDMA=0 # 已知 iGPU hang 源
PYTORCH_TUNABLEOP_ENABLED=0
TTS_AUTOCHUNK=false # 分块连续生成也会挂，一次性合成
TTS_LAZY_LOAD=false

HSA_ENABLE_SDMA=0 是社区已知的 iGPU 稳定性开关，TTS_AUTOCHUNK=false 是因为实测自动分块的多段连续生成也会触发 hang，关掉后长文本一次合成。

性能与稳定性验证

修复后的 benchmark（1.7B + sdpa + fast codebook，无 torch.compile）：

测试	生成耗时	音频时长	RTF
英文短句	6.8s	2.9s	2.32
英文中长句	27.4s	11.9s	2.32
中文中长句	20.0s	8.6s	2.32
整体 p95	—	—	2.34

RTF 2.32 意味着生成 1 秒音频要花 2.3 秒。对 780M 集成 GPU 这是合理水平：批量合成完全可用，实时对话（需要 RTF < 1）不行。

真正让人松一口气的是长文本验证：合成《蜀道难》全文 500+ 字，一次性生成 119.8 秒音频（24kHz/16bit WAV），耗时 5 分 38 秒，全程无 hang， GPU 稳定满载。之前这种长度是 100% 必挂的。

顺带发现一个数据陷阱：本地标着 “0.6B” 的模型目录其实是 1.7B 的完整拷贝（两个文件 MD5 完全相同，config 里 tts_model_size: "1b7"）。真 0.6B （905M 参数）在另一处旧部署目录里。换模型时不能只看目录名。

对未来 LLM / MoE 部署的经验清单

先验证镜像 tag 存在，再写 Dockerfile。Docker Hub 的 tag 列表可以直接翻页查询，5 分钟能省掉一次构建失败。
ROCm 镜像里锁死所有 torch 相关依赖版本，裸依赖必被 pip 解析成 CUDA 最新版。统一用 +rocm<ver> 后缀 + 官方 ROCm wheel 源。
gfx1103 一律 HSA_OVERRIDE_GFX_VERSION=11.0.0（伪装 gfx1100）。 11.0.2/11.0.3 是 gfx1103 真实版本号，但镜像没有对应 kernel，会报 invalid device function。这个值在 compose 里设置，覆盖 Dockerfile ENV。
在 APU 上禁用 PYTORCH_TUNABLEOP_ENABLED。kernel 调优基准会挂集成 GPU，这是最容易忽略的隐藏炸弹。
禁用 flash-attn 与 AOTriton flash SDPA。flash-attn 编译能成功但运行必挂；SDPA 用 sitecustomize.py 强制 enable_flash_sdp(False)， Python 启动即生效。环境变量 TORCH_SDPA_ENABLE_FLASH=0 在 ROCm 上不可靠，且测试要用 bf16 输入否则是假阳性。
警惕 transformers generate() 的逐 token .item() 同步。任何在生成循环里做 GPU→CPU 同步的代码，在 780M 上都是定时炸弹。优先选无同步的生成路径（本项目是 fast codebook），或把结束判断放到 GPU 侧。
HSA_ENABLE_INTERRUPT 保持默认，不要设 0。轮询模式把信号丢失变成 CPU 自旋，症状更难判断。
诊断三件套：py-spy 抓 Python 栈、gdb 抓 C 栈、gpu_busy_percent 盯 GPU 负载。三者交叉能快速区分"GPU 算不动"和"CPU 等不到”。
测试用例要覆盖长度梯度：短文本成功不代表长文本能过，信号丢失是概率问题，同步次数越多越容易触发。
持久化三件套：restart: unless-stopped、模型与配置 bind mount、缓存目录（~/.cache/miopen、torch_extensions）bind mount，否则容器 recreate 后 kernel 重新搜索编译，可能重现 hang。
GPU 频率锁 high 能提升稳定性：/sys/class/drm/card0/device/ power_dpm_force_performance_level 写 high 需 root 且重启失效，用 systemd oneshot 服务持久化。
验证模型真实性：目录名、config 的 model_size 字段、文件 MD5 三者对照，别被复制品误导。

Boundary conditions

结论基于 ROCm 6.4.3 + torch 2.6.0 + Ubuntu 24.04 这个组合。换 ROCm 版本（尤其 7.x）后，kernel 覆盖和 AOTriton 行为都会变，HSA override 值可能需要重测。
gfx1103 伪装 gfx1100 只是兼容性手段，性能以实测为准；不同 APU （如 gfx1101/1102 或 Strix Point）可能有不同的稳定参数组合。
fast codebook 绕过的是本项目 code_predictor 的 HF generate()。LLM / MoE 推理如果必须用 transformers generate()（如采样逻辑复杂、需要 stopping criteria），这个方案不能直接平移，需要另找无同步路径。
RTF 2.32 是"能用"不是"好用”。流式端点、voice clone、Base 模型（非 CustomVoice）在本部署中未验证，可能仍会触发 hang。
短文本 2.6s 的成功在早期版本里是侥幸（同步次数少），不能作为 “某个配置有效"的证据，必须以中长文本复测为准。

Open questions

0.6B 真模型（905M 参数）在相同配置下 RTF 能到多少？如果逼近 1.0，实时场景是否可行？
torch.compile 在 780M 上是否有可用配置（例如 reduce-overhead + 手动 warmup）？如果能用，RTF 2.32 还有多少下降空间？
ROCm 7.x 在 gfx1103 上的 SIGSEGV 是否已被修复？如果修复，是否值得迁移以获得更好的 kernel 覆盖？
流式输出（/v1/audio/speech 的 streaming 模式）在 fast codebook 下是否稳定？这决定 voice agent 场景的可行性。
对 LLM / MoE 推理（如 vLLM 的 ROCm 支持），HSA override 和 tunableop 的经验能否直接复用，还是 vLLM 有自己的 APU 参数体系？

[[Q]] 半年后重读：780M 上有没有一个完全无 .item() 同步的生成路径（包括自定义 stopping criteria），能让 transformers generate() 在 APU 上稳定跑长序列？

References

Qwen3-TTS-Openai-Fastapi 项目仓库（Dockerfile.rocm / config.yaml / api/backends/optimized_backend.py），2026-08 本地副本。
Docker Hub rocm/pytorch 镜像 tag 列表，2026-08 查询。
Qwen3-TTS-12Hz-1.7B-CustomVoice 模型 config.json（本地模型目录）。
PyTorch ROCm wheel 索引 download.pytorch.org/whl/rocm6.2.4，2026-08 访问。
AMD ROCm 社区对 gfx1103 与 HSA_OVERRIDE_GFX_VERSION 的讨论（网络搜索， 2026-08）。

AI 怎么想问题：从数学推理漫游中读到的认知画像

Sun, 02 Aug 2026 00:00:00 +0000

引言：一个模型在解释另一个模型的想法

我花了一个晚上读 OpenAI 发布的数学推理漫游系列（“How the Ideas Came Together”），十二个章节，覆盖非 sofic 群构造、Connes 刚性反例、量子并行重复定理、多重色 Ramsey 数、Erdős–Simonovits 紧致性反例等十个数学问题领域。每一章不是论文复述，而是把"想法是怎么拼出来的"讲成一个故事：哪些想法先指了路，哪些大路撞了墙，什么视角转换揭开了结构。

打动我的不是那些定理本身，而是这个元设：写这些笔记的"作者"是一个 AI 模型，它读了原始 chain-of-thought 和最终论文，然后回头重建推理过程的演化史。

这篇笔记记录我从这套漫游中读到的 AI 认知画像，以及我对一个问题的持续困惑：当 AI 用第一人称叙述自己的推理时，那是理解的雏形，还是一种精致的 confabulation？

Scope：本文覆盖漫游系列中反复出现的认知模式，不覆盖任何定理的完整证明，也不评价 OpenAI 与人类数学家的相对能力。

Prerequisites：假设你了解 chain-of-thought 的基本概念，且对数学证明的形态有直觉。群论、凸几何、极值图论的细节我会在用到时给出足够上下文。

叙事装置：三层认知结构

漫游系列的构造本身就是一个三层结构：

第一层：AI 解题时留下的原始思维流（chain-of-thought），混沌、有噪声、充满死胡同。
第二层：AI 事后对思维流的叙事化重述，把曲折路径整理成"哪条路先想到、哪条路撞墙、什么转换揭开结构"的因果故事。
第三层：我们人类读第二层。

三层之间不是忠实转录。第二层是重建，不是回放。它预设了一种认知观：想法不是灵光一闪，而是"路径 + 障碍 + 换视角"的演化。这个预设本身就是一种立场，它把数学发现描述成可叙述的过程，而不是不可言说的顿悟。

每一章遵循同一骨架：哪些想法先指了路、哪些大路撞了墙、什么视角转换揭开了结构、决定性洞见怎么长成最终论证。骨架如此一致，以至于我怀疑它部分地是在为读者塑造一种"理想的推理形态"，但这恰恰是它最有趣的地方：即便有粉饰，被选进叙事的事件仍然透露了底层搜索的真实形状。

贯穿十二个章节，我辨认出五个反复出现的认知模式，下面逐一展开。

失败是主要的认知工具

最常见的假设是失败是浪费。这套漫游给出的证据相反：几乎每个突破都由结构性失败铺路，而且失败的方式高度一致：它排除一整类机制，把搜索逼向正确的方向。

Ehrhart 不等式章节（ch09）是典型。harmonic symmetrization 路线走了很久：它能精确命中极值对象 $T_n$：普通交集在二维就丢体积，对称化后体积恰好是 $2^n$，却解释不了缺失的 $n!$。结论写得极其诚实：“它识别了极值对象，却没有解释它的阶乘。“这条死路没有白走：它排除了"凸几何对称化"整类机制，把搜索逼向真正 n 维的计数机制，最终落在 jet 计数上。

量子并行重复章节（ch07）把失败用得更系统。连续四类约减（投影游戏张量化、anchoring、oracularization、顺序熵累积）全部失败，且失败原因同构：都改变了物理策略类，而不是量化损失。原文直接说这些重复失败把搜索从普通信息界推向纯化规范。失败在这里不是噪音，是搜索方向修正器。

非 sofic 群章节（ch03）的绕路更有名：想把 property (T) 直接转成 mixing，死于二部图谱可以接近 $-1$ 而 Kazhdan 间隙在 $1$。核心错配被明确命名：“Kun 给出许多 expander，Kun–Thom 只需要一个。”

我原先以为 AI 的失败记录是事后修饰出来的叙事点缀。读完全部章节后我改了这个判断：失败的出现频率、具体程度和后续引用（“正是这次失败告诉我们 X 行不通”）太一致了，更像是真实搜索过程的压缩回放。失败不是装饰，是认知工具。

显式反例杀死自己的猜测

漫游中最让我意外的一类行为：AI 会构造小反例来证伪自己的中间猜测，而不是绕过它。

最漂亮的是永久型下界章节（ch02）里的 Krawtchouk 事件。第一个二元递推基于一个 associated Krawtchouk 猜测。AI 自己构造了 $n = 8$、原始度 $k = 1$ 的反例：猜测值 $508/7 < 128$，而真实码有 128 个码字。猜测被自己的反例直接证伪。

这不是试错，这是数学家的习惯：对中间猜测做最小实例的假说检验。而且修正不是打补丁，是结构性的：正确项把整个分子移出平方根。反例指出的不是参数偏差，而是公式形态的错误。

同样的事件出现在球堆积章节（ch01）：全局范数比较路线被显式反例堵死：一个函数可以有大的全局范数比，却不把负质量放进禁区半径。障碍不是未优化的常数，而是信息层面的缺陷：全局范数忘了负质量在哪。这类反例的价值在于它精确定位了论证中"哪条信息被丢弃了”，而这正是换坐标系（下一节）的动机。

视角转换：换坐标系是核心武器

每一个突破都伴随一个"换坐标系"的动作。漫游里这个模式如此频繁，我已经把它当作 AI 推理的主干操作。

球堆积章节：径向 Fourier 变换变成 Mellin 反射加显式相位，谐和测度在带的边缘识别出精确半径 $1/\pi$。高斯提供正确的 Fourier 对称性但错误的鞍点位置，于是变形加 remote shell 修复。

Connes 刚性章节（ch04）的决定性转变是一个问题：“crossed product 到底记得什么？“答案：它记得概率空间、Haar 测度和可测作用，但不记得对偶上的紧群律。于是目标从"找两个同构 K-模"变成"同一个 K 概率空间上的两种群律”。可测共轭与代数共轭的区分组织了整个构造。特征二在这里是主题性的：二次布尔进位 $B = \mathrm{span}{v \otimes v}$ 对可测作用不可见、对离散对偶可见，于是"同一个概率空间上两种紧群律"就够了，还能平移出可数无穷纤维，指数 $[\Gamma_n : \Gamma_0] = 2^{4n}$。注意原文有一句 “This choice took some work”。divided square 不是第一次就选对，普通对称平方商会模糊特征二极化。这个选择本身是迭代出来的。

Ehrhart 章节的换坐标系最激进：“rationality 不必要”：任意凸体都能有 toric 势（实 Monge–Ampère 定理），于是扔掉有理多胞形的包袱，在非紧复环面上工作。

一个统一的观察：这些转换不是随机的。它们都由前一轮失败的"信息缺陷"定向：哪条信息被丢了，就往能重新看见那条信息的坐标系里跳。失败定位缺陷，转换修复缺陷。两件事是一套循环。

从失败中提炼元规则，并守恒不变量

第三章有一段教科书级的元认知陈述，我读到时愣了一下，因为这种"从失败里抽规则"的行为正是我写论文时最羡慕的能力。log-midrank 中间方案失败后，身份等式揭示了原则：

$$\mathrm{Var}(\mathrm{midrank}) = \frac{1 - \sum_j p_j^3}{12}$$

这个等式（其中 $p_j$ 是组件大小分布）被读作一条原则：“平均组件大小的有界单调函数，绝不平均无界的大小本身。“最终方案，顶点加权中位数 $f = M / (M + m_A)$，就是这条原则的干净实现。AI 不仅记录做了什么，还记录从失败中抽象出的"为什么这样做"的规则。这已经是 Polya 式启发法的形态。

贯穿全书的是"什么东西必须精确保持"的自觉。量子并行重复章节保 Born 权重：共轭对恒等式 $F^{1/2+it} \cdot F^{1/2-it} = F$ 和有限预解纯化。永久型下界章节的正性来自真实 Gram 余项恒等式，而不是"假定 associated 多项式为正”。球堆积章节保符号位置而非全局范数。非 sofic 群章节把组件尺寸匹配到 $\rho_n \to 1$，运输分割被保全。

这跟物理学的守恒量直觉同构：先找到论证中必须不变的量，失败模式往往就是某个量被破坏了。AI 的搜索看起来像是对"不变量"的持续探测：每次失败都问一句：刚才什么量变了？

显式有限见证优于渐近启发

漫游偏爱"能指着一个对象说这就是为什么"的论证，而不是"平均来说应该可以”。这个偏好很具体：

remote shells：可显式构造的球堆积修正层，比渐近密度论证更受信任。
饱和矩阵：来自 Alon–Ben-Eliezer–Shangguan–Tamo 帽子猜测的显式结构。
symplectic 广义四边形 $W(q)$：作为稠密见证的有限几何对象。

最极端的例子在第 12 章：2-退化图超指数反例。熵窗口 $\kappa + \tau \log_2 3 < \beta < 2h(\tau) - 1$ 在 $\tau = (\sqrt{3}-1)/2$ 处优化，间隙 $w \approx 0.003728$——窄但严格为正。原文的表述我记了很久：“一旦为正，大小就不再是障碍。“它知道正性比幅度重要。一个 $0.0037$ 的间隙和 $0.5$ 的间隙在存在性论证里等价。

这个偏好与人类数学家的习惯一致：构造性证明比存在性证明更容易被社区核查、简化和复用。AI 似乎学到了这一点，或者被训练数据里的数学文化塑造了这一点。

局限：叙述不是验证

漫游系列自己写了 Limitations，而且没有美化：这是 AI 的事后回顾，“反映的是被记录的推理过程，而非对每一步的独立验证”。

这个限定我必须认真对待。回顾性叙事有 hindsight bias 的风险：曲折路径被整理成连贯故事，可能高估了"计划性”、低估了随机搜索。我在上面总结的"失败定位缺陷，转换修复缺陷"的循环，可能是真实搜索的样子，也可能是叙事把随机撞墙整理成了因果链条。

还有几件事值得记住：没有任何结果声称被机器验证；GapCVP 章节的指数大到 $O(N^{802})$ 位，只保多项式时间可计算性；多重色 Ramsey 的常数 $6e^{38}$ 是插值吸收掉的；Ehrhart 等式分类仍开放。这套漫游呈现的是"AI 认为自己在做什么”，不是"AI 证明了什么”。

更深一层的张力在这里：如果把推理过程的叙述当作理解的定义，那么 AI 拥有理解；如果理解要求对每一步的独立可验证把握，那么叙述只是 product 的另一种形式。Park 的表述我一直觉得锋利：“proof is a product, understanding is a capacity。“漫游系列把 product 翻译成 capacity 的语言：它让黑箱推理变得可以被人类社区检查、简化、吸收。它不能替代理解，但它是在为"可理解性"争取空间。

Boundary conditions

这套认知画像的边界，我目前看到这些：

它基于单一来源（OpenAI 的漫游系列），样本是十二个精心挑选的成功案例。失败到完全放弃的案例不在其中，搜索的真实失败率未知。
事后叙事可能系统性高估计划的连贯性。没有独立于叙事的过程数据，无法区分"结构发现"与"事后归因”。
我的解读本身是第三层阅读（读 AI 的重述），可能放大了漫游刻意塑造的"理想推理形态”。
“换坐标系"模式可能被过度概括：它确实是高频模式，但我没有统计它的覆盖率。
我不知道这些模式在非数学领域（编程、科学实验设计）是否同样成立。这套画像可能只是数学推理的特例。

Open questions

如果给 AI 同一个问题的两套独立 chain-of-thought，事后叙事会收敛到同一套"认知模式”，还是各自编造不同的故事？这能检验叙事是忠实压缩还是 confabulation。
失败排除机制类的能力，在多大程度上来自训练数据里的数学文化，多大程度上是 RL 搜索自然涌现的？
保不变量直觉是否有一个可操作的算法形式？比如"检测论证中被破坏的量"能否自动化成调试工具？
人类数学家读漫游后能加速理解定理；如果 AI 生成面向自身的"推理地图”，它对 AI 自身的后续搜索有反馈价值吗？
六到十二个月后重读：当更多独立证据出现时，我还会认为失败是 AI 的主要认知工具吗？

[[Q]] 六个月后：如果出现独立于 OpenAI 的 AI 数学推理过程记录（比如开源模型的长思维链日志），我总结的五种认知模式（失败排除、反例证伪、换坐标系、不变量守恒、显式见证）中有几种能复现？

References

OpenAI, “How the Ideas Came Together: Notes on AI Mathematical Reasoning”（十二章节漫游系列）， https://cdn.openai.com/pdf/reasoning-walkthroughs.pdf。
Kun, G. 与 Thom, A.，关于 non-sofic groups 与 expanders 的工作（ch03 引用）。
Cohn, H. 与 Elkies, N.，球堆积上界理论（ch01 背景）。
Erdős, P. 与 Simonovits, M.，极值图论紧致性猜想（ch11 背景）。
Alon, N.、Ben-Eliezer, T.、Shangguan, C. 与 Tamo, I.，帽子猜测与饱和矩阵（ch12 引用）。
Park,Automation Without Understanding， https://arxiv.org/abs/2607.06377。

From engineered engagement to evidence-based development: the neuroscience of short-form video and a framework for redirecting its mechanisms

Tue, 14 Jul 2026 00:00:00 +0000

I kept encountering the same paradox in the literature on short-form video. The platforms are described as “addictive,” yet population-level harm effects are tiny. The mechanisms are called “exploitative,” yet they map almost perfectly onto established findings in learning science. And the proposed solutions oscillate between moral panic and naive techno-optimism. I wanted a single, honest account that holds both sides without collapsing into either.

The central claim of this article is that short-form video engagement is driven by three converging mechanisms — dopaminergic prediction error, variable-ratio reinforcement, and algorithmic personalization — whose effects are substrate-neutral: they can serve compulsive consumption or structured learning depending on design intent and guardrails. The practical consequence is that the same features making platforms compelling can be ethically redirected toward mastery, provided we replace the reward target from “next clip” to “next competence signal.”

Scope. This covers the psychological and neurological mechanisms of short-form video engagement, the contested evidence for “addiction,” and a practical framework for redirecting these mechanisms toward learning and habit formation. It does not cover clinical diagnosis, platform engineering internals, content moderation, or legislative policy.

Prerequisites. This assumes familiarity with basic dopamine reward systems, operant conditioning, and cognitive load theory.

Part I: The three engines of engagement

The neural reward engine

At the foundation of short-form video’s pull is a single, well-replicated finding: dopamine neurons encode reward prediction error. They fire strongly when a reward is better than expected, fire to cues that predict reward, and remain silent when outcomes are fully predicted. This was established through classic electrophysiology — Schultz (1998) recorded from midbrain dopamine neurons while monkeys learned associations between visual cues and juice rewards, finding that dopamine firing shifted from the reward itself to the predictive cue as learning progressed.

The significance for short-form video lies in a related result: dopamine responses also scale with reward uncertainty, rising most when the probability of reward is intermediate. A TikTok feed is precisely this condition. The user cannot predict whether the next clip will be highly entertaining, mildly interesting, or entirely flat. The brain is maintained in a state of maximum anticipation, and the uncertainty itself is the engine.

This connects to a distinction that matters enormously for understanding compulsive use: the dissociation between “wanting” (incentive salience, driven by mesolimbic dopamine) and “liking” (hedonic pleasure, which does not depend on dopamine). The felt experience of “I don’t even enjoy this but I can’t stop” is a direct prediction of this model. The wanting system can run independently of the liking system, and variable-ratio reward schedules are precisely the conditions that drive it hardest.

A parallel system, less discussed but potentially important, operates through Action Prediction Error (APE). While reward prediction error evaluates outcomes and drives learning about value, APE neurons in the tail of the striatum track how often an action is performed, serving as a value-free teaching signal that consolidates habitual motor behavior. The automatic thumb-swipe — the physical gesture of scrolling — may be solidified through this mechanism, creating a habit loop that is neurologically resistant to conscious interruption even when the user recognizes the behavior is unproductive.

The behavioral schedule

Beneath the neuroscience sits a classic behavioral structure. A variable-ratio reinforcement schedule — reward delivered after an unpredictable number of responses — produces the highest, most persistent response rates and the greatest resistance to extinction of any partial-reinforcement schedule. This is operant psychology 101, established through decades of work with animal models and replicated across species.

Scrolling a feed in which only some clips are rewarding is functionally a variable-ratio schedule. The pull-to-refresh gesture or the swipe-up is the behavioral trigger. Because the cost of each response is nearly zero (a thumb movement) and the payoff is uncertain, the system generates the maximum possible rate of responding. Habit-forming product design has made this explicit: designers deliberately apply variable-reward schedules modeled on the operant conditioning literature.

What makes the digital version particularly potent is the combination of near-zero response cost with algorithmic improvement of hit rate. In a traditional variable-ratio schedule (say, a slot machine), the reward probability is fixed. In a short-form feed, the recommender system learns from each interaction what the user finds rewarding and progressively increases the probability of a “hit.” The schedule is not merely variable — it is adaptive, getting better at predicting what will keep the user engaged.

The algorithm as amplifier

Short-video recommender systems learn user preferences primarily from the implicit watch-time signal and personalize the feed rapidly, without requiring explicit ratings. In a donated-data study, participants’ daily videos viewed and time on platform roughly doubled within about 80 days. The precise doubling figure comes from a single study and should be read as indicative rather than a general effect size, but the qualitative loop is well-documented: the more a user watches, the better the model predicts what will hold attention, which increases watching.

At the neural level, personalized recommendation algorithms are effective at up-regulating activity in both the ventral tegmental area (VTA) — the origin of dopaminergic cell bodies — and sub-regions of the default mode network (DMN). Specifically, viewing personalized content increases coupling between the posterior cingulate cortex and sensory cortices (deepening sensory immersion) while decreasing coupling between the medial prefrontal cortex and regions involved in cognitive evaluation. The algorithm is not merely selecting content; it is altering the neural conditions under which the user processes information, reducing the brain’s capacity for critical assessment while amplifying sensory engagement.

This creates what I think of as the core tension of the entire problem: the same algorithmic personalization that makes content maximally engaging also makes it maximally difficult to disengage. The system learns to exploit individual cognitive vulnerabilities — not through malice, but through optimization of a simple objective function (maximize watch time) applied to a substrate (the human reward system) that was not designed for this environment.

Part II: The costs — and the contested question of harm

Attention and its measurable costs

The novelty and switching that make the feed engaging carry attentional costs. Heavy media multitasking and frequent short-form video use are associated with weaker attentional filtering, larger task-switching costs, and more frequent attentional lapses with poorer incidental memory.

EEG studies provide more specific evidence. A study using the Attention Network Test found a significant negative correlation (r = -0.395, p = 0.007) between short-video addiction tendencies and theta wave power in frontal electrodes during cognitive conflict resolution. Theta oscillations in the frontal cortex are essential for recruiting neural resources to manage competing signals and exert executive control. The critical detail: this neural degradation occurred even in the absence of observable behavioral deficits on the task, indicating a “neural masking” effect where executive control signals atrophy before performance drops become visible.

The broader brainwave picture is consistent. Gamma power increases by 40-62% during high-reward moments (indicating hyper-arousal from rapid context-switching), prefrontal beta power reduces by 22% after just 20 minutes (reflecting impaired decision-making), alpha declines during engagement (suggesting unsustainable cognitive load), and delta increases by up to 50% over two years in heavy users (correlating with digital fatigue and degraded sleep architecture).

I find the “neural masking” result particularly important. It means that by the time someone notices their attention is degraded, the underlying neural substrate has already been changing for some time. The behavioral symptom is a lagging indicator.

Memory fragmentation

A consequence I did not expect to find in the literature: rapid short-form consumption may actively impair the brain’s ability to process continuous information. Human cognition relies on event segmentation — parsing continuous experience into discrete, meaningful units. When a prediction fails, the brain establishes an event boundary and constructs a new predictive model.

Short-form video forces artificial prediction errors at extreme frequency (every 15-30 seconds), habituating the brain to a “segment-and-refresh” mode. Eye-tracking studies using Hidden Markov Models show that people exposed to random short videos exhibit significantly more fragmented perception. Neuroimaging data makes the cost concrete: learning from fragmented short videos drops memory retrieval accuracy from approximately 66.4% to 43.3%, with reduced activation in the left claustrum (cognitive control and multisensory integration), left caudate nucleus (goal-directed behavior), and left middle temporal gyrus (semantic processing), plus broken functional connectivity between caudate and claustrum.

The implication is that short-form video does not merely waste time. It may train a mode of information processing that is actively hostile to the kind of sustained, integrative thinking that deep learning requires. When novelty comes as relentless context switching, it supports orienting and stimulation while undermining semantic integration, elaboration, and later retrieval.

The sleep connection

Sleep contributes to memory consolidation through hippocampo-thalamocortical synchronization during sleep cycles. Sleep deprivation impairs encoding and retention of newly learned material. Bedtime scrolling is concerning not only because it displaces total sleep time, but because it may turn potentially useful learning episodes into poorly consolidated ones. The hippocampus is central because short-form content often creates the illusion of learning: high familiarity and high stimulation can be mistaken for durable memory, but episodic and semantic memory formation depend on consolidation processes that sleep supports.

How much of this is “addiction”?

The clinical picture is genuinely mixed, and I want to hold both sides visible. On one side, validated short-video and TikTok dependence scales exist and, in a 16,038-person study, identify a dependent subgroup of roughly 7.5% (plus about 16% at-risk), converging with DSM-5-style dependence criteria. Problematic use is further associated with measurable attention deficits and with elevated depression, anxiety, and stress.

On the other side, the proposition that short-form video constitutes a clinical behavioral addiction comparable to established disorders is contradicted by rigorous, pre-registered, large-sample analysis concluding that the association between digital-technology use and adolescent well-being is negative but tiny — explaining at most about 0.4% of variance across datasets totaling more than 350,000 participants, and “too small to warrant policy change.”

The honest synthesis is that a minority show dependence-like patterns while population-level harm is small and confounded. This matters for the framework: harm-reduction guardrails should be proportionate and targeted, not framed as if universal pathology were established. The defensible design principle is to build systems that work well for most people while providing guardrails for the vulnerable, rather than assuming everyone is at risk.

Part III: Turning the engine around

The alignment between engagement and learning science

Here is where the story gets interesting. The features that make short-form video compelling map onto some of the most robust findings in learning science, but with a critical inversion.

Mechanism in SFV	Learning science parallel	Key difference
Short segments	Segmenting effect: breaking instruction into learner-paced segments improves retention and reduces cognitive load	SFV segments are arbitrary (entertainment); learning segments are pedagogically meaningful
Novelty variation	Novel stimuli enhance memory encoding via dopaminergic hippocampal modulation	SFV novelty is relentless context switching; learning novelty should be bounded and schema-integrative
Continuous feedback	Immediate feedback supports competence satisfaction and motivation	SFV feedback is social validation (likes); learning feedback should be informational (mastery signals)
Personalization	Adaptive difficulty maintains flow state	SFV personalization maximizes watch time; learning personalization should optimize retention

The correspondence is real but partial. The segmenting effect, the spacing effect, and the testing effect are among the most replicated findings in educational psychology. Short-form video already delivers short segments, novelty variation, and immediacy — so the format is a natural vehicle for structured learning. But the default configuration points the wrong way: passive consumption rather than active retrieval, entertainment reward rather than competence reward, infinite continuation rather than bounded sessions.

Self-Determination Theory as the replacement engine

Self-Determination Theory (SDT) identifies three basic psychological needs whose satisfaction fosters intrinsic motivation: autonomy (the need for control), competence (the need for mastery), and relatedness (the need for connection). In interactive technology specifically, competence-satisfying feedback and masterable, autonomy-supportive design predict enjoyment and sustained motivation.

This matters because it names a healthy engine of engagement that can substitute for the variable-ratio hook. The variable-ratio schedule drives behavior through uncertainty and external reward. SDT-aligned design drives behavior through agency and internal satisfaction. They are not the same mechanism; the “repurposing” is partly a substitution.

A landmark meta-analysis found that tangible, contingent extrinsic rewards can undermine intrinsic motivation (d = -0.28 to -0.40), whereas positive informational feedback enhances it (d = 0.31 to 0.33). This single result carries much of the ethical weight of the framework: it is why the recommendation is to move away from variable-ratio hooks toward competence feedback. Points, badges, and leaderboards that function as controlling external rewards can backfire. Progress visibility, skill trees, and narrative-embedded mastery signals that function as informational feedback support sustained engagement.

The learning science stack

The component evidence for redirecting engagement toward learning is strong at the individual mechanism level, even if integrated systems have not been directly validated.

Spacing and retrieval. Distributing practice over time (the spacing effect) and actively retrieving information (the testing effect) are among the most robust enhancers of long-term retention. The optimal inter-study interval depends on the desired retention interval — roughly 10-20% of the target retention period. A learning system should present micro-units at expanding intervals, exploiting the brain’s natural consolidation processes.

Segmenting and microlearning. Breaking instruction into short, learner-paced segments improves retention and transfer while lowering cognitive load. Microlearning applies this principle with measurable outcome gains. The critical qualification: brevity alone is not instructional quality. Short clips improve access and initial uptake, but durable learning requires retrieval, spacing, and transfer practice rather than mere exposure.

Interleaving. Studying different but related concepts within a single session, rather than one concept exhaustively, introduces desirable difficulty that improves transfer and critical thinking. This actively counters the rigid over-segmentation caused by recreational short-form video.

From engaged sessions to durable habits

Habit automaticity forms gradually through consistent, context-dependent repetition, taking about 66 days on average (range 18 to 254) with occasional missed repetitions not derailing the process. Forming an implementation intention — an explicit if-then plan linking a cue to an action — substantially increases goal attainment (d = 0.65 across 94 tests). Behavior occurs when motivation, ability, and a prompt converge (Fogg’s B = MAP model).

The practical design rule is that simplicity changes behavior more effectively than motivation. Because motivation is an unreliable, fluctuating emotional state, relying on it for skill acquisition leads to failure. The target behavior must be reduced to an almost absurdly simple action (under 30 seconds), anchored to an existing routine as a cue, and followed by self-generated positive reinforcement to trigger a dopamine response that consolidates the behavior as rewarding.

Part IV: The applied framework

The central design rule

The framework rests on one principle: preserve the attentional efficiency of short-form content while replacing compulsive continuation with bounded mastery cycles. This means keeping the short segments, the immediacy, and the personalization while changing the reward target, the feedback structure, and the session architecture.

Step-by-step

One micro-objective per clip. Each short unit should answer exactly one question or train one action: one theorem idea, one pronunciation contrast, one coding pattern. This follows the evidence behind microlearning, segmentation, and cognitive-load reduction.

Front-load curiosity, not clutter. Open with a concrete problem, prediction prompt, or noticeable error rather than sensory overload. Novelty helps when it signals relevance; too much surface novelty risks fragmented processing.

Force an action before the answer appears. Insert a prompt to retrieve, predict, imitate, or classify before the reveal. Retrieval practice generally outperforms passive review, and even simple retrieval prompts can improve delayed problem solving. This is where the variable-ratio schedule gets replaced: instead of “maybe the next clip will be good,” the uncertainty becomes “can I get this right?”

Use bounded reinforcement. Provide an immediate success signal after correct retrieval or execution, but avoid endless variable-ratio continuation. Good rewards are progress bars, level completion, skill streaks, or supportive peer feedback tied to effort and mastery. Not “one more clip.”

Schedule reactivation instead of bingeing. Deliver the next clip in a spaced sequence rather than an infinite feed: same-day recap, next-day retrieval, then expanding intervals. The algorithm that currently personalizes for watch time should personalize for retention.

Bind learning to a stable cue. Attach the short learning unit to a reliable context such as “after coffee,” “after opening the laptop,” or “immediately after lunch.” Habit research consistently shows that context stability and cue-based enactment build automaticity more effectively than vague intention alone.

Close the loop off-screen. Every cluster of clips should end in a non-feed action: speaking a phrase aloud, solving one practice item, performing one movement, writing one summary sentence, or teaching the concept to someone else. This helps convert familiarity into transfer.

Protect consolidation. Use hard stopping rules, especially before bedtime. Sleep is not an optional extra for learning; it is part of the memory system. Short-form educational routines should include no-autoplay rules, session caps, and a pre-sleep buffer.

Ethical guardrails

An ethical short-form educational system should be optimized for mastery per minute, not minutes per user. That distinction is essential because the same mechanics that increase retention of users can decrease retention of knowledge.

The specific risks to manage: infinite continuation (remove infinite scroll and autoplay in educational contexts), vulnerable-user amplification (default-friction safeguards for youth, high-FoMO users, low self-control users), popularity over pedagogy (emphasize mastery badges over raw popularity counts), shallow learning masquerading as productivity (require delayed quizzes and off-screen transfer tasks), sleep displacement (no reminders near bedtime), and opaque personalization (make recommendation logic legible; let learners choose playlists, pacing, and stop points).

Evaluation metrics

Because the literature warns against using time-spent alone as the core metric, evaluation should separate engagement quality, learning quality, habit quality, and well-being cost. Completion rate, voluntary rewatch rate, and return rate distinguish active engagement from passive exposure. End-of-clip accuracy and one-step problem solving capture immediate uptake. Twenty-four-hour and seven-day recall, cumulative quiz performance, and transfer tasks prevent the familiarity illusion. Habit automaticity scores and context-stable enactment rates track whether behavior has become automatic. Attention-control measures, mind-wandering reports, and time-to-re-engage deep work detect whether the intervention is eroding control. Session length variance, bedtime use, sleep quality, negative affect, and academic displacement catch “engagement wins” that are actually developmental losses.

Boundary conditions

The framework assumes content that can be meaningfully chunked into micro-units. Subjects requiring extended derivation, sustained argument, or immersive practice (e.g., advanced mathematics, complex surgery, musical performance) may not adapt well to short-form delivery.
The learning-science components (spacing, retrieval, interleaving) are individually well-validated, but their integration into a single short-form system has not been directly tested as a unified package. The framework is a synthesis of supported analogies, not a validated product.
The addiction question is unresolved by design. The framework targets guardrails for the vulnerable rather than assuming universal pathology, but this calibrated approach may underserve individuals with genuine compulsive-use disorders who need clinical intervention, not better product design.
Most of the evidence base is drawn from adolescents and university students, with heavy representation from Chinese samples. Cross-cultural and adult generalization is uncertain.
The framework does not address the political economy of platform design: the business model that funds algorithmic personalization is advertising revenue maximized by watch time, and “mastery per minute” optimization conflicts with that incentive structure.
Individual differences in impulsivity, self-control, attachment style, and baseline mental health moderate all of these effects. A framework that works for a typical user may fail for someone with high trait anxiety or low executive function baseline.

Open questions

Does the Action Prediction Error mechanism (value-free habit consolidation in the tail of the striatum) actually apply to the thumb-swipe gesture, or is it specific to goal-directed motor sequences? The APE research is recent and mostly from animal models.
What is the actual dose-response curve for fragmented viewing and memory impairment? The 66.4% to 43.3% accuracy drop comes from a single fMRI study with specific materials. Does this generalize across content types and populations?
Can algorithmic personalization be repurposed for retention optimization without recreating the compulsive-use dynamics it currently serves? The technical challenge is that the same feedback loop that improves watch time also increases engagement — and disentangling these requires a different objective function.
How does attention restoration interact with the microlearning framework? The attention restoration literature suggests that directed attention requires recovery periods, but the optimal “dose” of restoration relative to learning-session length is not established.
The framework recommends replacing variable-ratio entertainment rewards with competence-contingent feedback. But does competence feedback activate the same dopaminergic circuits, or does it rely on different neural pathways? If different, the “redirecting engagement” metaphor is misleading — it is more like substituting one motivation system for another.

[[Q]] Six months from now: in any chance there is a randomized trial comparing a short-form learning system built on these principles (spaced micro-units, retrieval prompts, competence feedback, session caps) against both a standard short-form entertainment feed and a traditional long-form learning control, with delayed retention and transfer as primary outcomes?

References

Schultz, W. “Predictive reward signal of dopamine neurons.” Journal of Neurophysiology, 1998.
Fiorillo, C. D., Tobler, P. N., & Schultz, W. “Discrete coding of reward probability and uncertainty by dopamine neurons.” Science, 2003.
Berridge, K. C., & Robinson, T. E. “Liking, wanting, and the incentive-sensitization theory of addiction.” American Psychologist, 2016.
Ophir, E., Nass, C., & Wagner, A. D. “Cognitive control in media multitaskers.” PNAS, 2009.
Zannettou, S., et al. “Analyzing User Engagement with TikTok’s Short Format Video Recommendations using Data Donations.” CHI, 2024.
Jiang, A., et al. “Assessing Short-Video Dependence: Development and Validation of the Short-Video Dependence Scale.” JMIR, 2025.
Orben, A., & Przybylski, A. K. “The association between adolescent well-being and digital technology use.” Nature Human Behaviour, 2019.
Deci, E. L., & Ryan, R. M. “Self-Determination Theory.” Center for Self-Determination Theory, 2000.
Cepeda, N. J., et al. “Distributed practice in verbal recall tasks: A review and quantitative synthesis.” Psychological Bulletin, 2006.
Roediger, H. L., & Karpicke, J. D. “Test-enhanced learning: Taking memory tests improves long-term retention.” Psychological Science, 2006.
Rey, G. D., et al. “A Meta-analysis of the Segmenting Effect.” Educational Psychology Review, 2019.
Lally, P., et al. “How are habits formed: Modelling habit formation in the real world.” European Journal of Social Psychology, 2010.
Gollwitzer, P. M., & Sheeran, P. “Implementation intentions and goal achievement: A meta-analysis.” Advances in Experimental Social Psychology, 2006.
Fogg, B. J. “Fogg Behavior Model (B=MAP).” behaviormodel.org, 2019.
Deci, E. L., Koestner, R., & Ryan, R. M. “A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation.” Psychological Bulletin, 1999.
Yan, T., et al. “Mobile phone short video use negatively impacts attention functions: an EEG study.” PubMed, 2024.
Wei, M., et al. “Fragmented learning from short videos modulates neural activity and connectivity during memory retrieval.” ResearchGate, 2026.
Sherman, L. E., et al. “The Power of the Like in Adolescence: Effects of Peer Influence on Neural and Behavioral Responses to Social Media.” Psychological Science, 2016.
Guo, P. J., et al. “How video production affects student engagement: An empirical study of MOOC videos.” L@S, 2014.
Brodt, S., et al. “Sleep — A brain-state serving systems memory consolidation.” Neuron, 2023.
Singh, B., et al. “Time to Form a Habit: a systematic review and meta-analysis.” Health Psychology Review, 2024.
U.S. Surgeon General. “Social Media and Youth Mental Health.” Advisory, 2023 (updated 2025).

从提示员到系统架构师：Loop Engineering 的范式跃迁

Sun, 28 Jun 2026 00:00:00 +0000

引言：你不再是对话者了

2026 年上半年，AI 编码代理从命令行玩具变成了日常基础设施。Claude Code、OpenCode、Codex CLI、Cursor——这些工具不再只是"更聪明的自动补全"。它们开始自主地阅读代码库、提出修改方案、运行测试、迭代修复。

与此同时，一个微妙但彻底的转变发生了。那些从这些工具中获得最大产出的人，不再花更多时间写更好的提示词。他们花时间设计系统——系统自己决定下一步做什么。

Boris Cherny（Claude Code 的创造者）在 2026 年 5 月说了一句我觉得精准捕捉了拐点的话：“我不再给 Claude 写提示词了。我有一些 loop 在运行，而那些 loop 给 Claude 写提示词……我的工作是写 loop。”

这句话像一把刀切开了两个时代。在此之前，你的价值取决于你多擅长和模型对话。在那之后，你的价值取决于你多擅长设计不需要你参与对话的系统。

这篇文章记录的是我对这个转变的理解——它在 2026 年中期呈现的样子。如果你一年后重读，有些判断可能已经过时了。但结构性的东西，那些层次的划分、原语的识别、难的问题和容易犯的错——我猜它们会留得更久。

一个不成立的假设和一个正在发生的事实

我见过太多团队在 2025–2026 年的做法：雇一个人"专门调 prompt"，把这个人的产出绑定在每一次聊天窗口的对话质量上。这是一种很自然的想法——模型是你的对话对象，对话质量决定输出质量。

但这个假设在 2026 年中间某个时刻悄悄失效了。

失效的方式是这样的。想象两个工程师在使用同一个模型。工程师 A 打开聊天窗口，写了一条很长的 prompt，得到了一份不错的代码，然后关掉窗口。工程师 B 做了一个自动化工作流：一个 git hook 在 PR 创建时触发一个 agent，agent 阅读 diff、运行测试、检查代码风格、写 review 评论。如果评论被采纳，agent 还会修改代码并推送新的 commit。

工程师 A 可能写了更优雅的 prompt。但工程师 B 一周内 Review 的 PR 数量是 A 的几十倍——而且他在这期间还做了一堆别的事情。

这不是 prompt 质量的问题。这是抽象层次的问题。A 在第一个层次上和模型互动。B 设计了一个运行在更高抽象层次上的系统，这个系统自己知道什么时候需要和模型对话。

我最初看到这个差距时，觉得它只是一个效率优化——让机器做更多，人做更少。后来我发现这不仅仅是效率。它改变了你作为工程师的界面。你的工作不再是"写更好的指令"，而是"设计更好的迭代结构"。你从一个操作员变成了架构师。

这个转变我目前认为不可逆。不是因为工具会越来越好——而是因为一旦你体验过设计一个 loop 然后看着它自动完成过去需要你逐轮介入的工作，你就很难再回到聊天窗口里了。

什么是 Loop Engineering

一个直觉

Loop Engineering 的定义很简单：设计自主迭代循环的实践。这些循环决定什么时候调用模型、传递什么上下文、怎么评估输出、什么时候重试、什么时候停止。

但它真正反直觉的地方是这个：loop 把工程师从循环内部移到了循环外部。

在 prompt engineering 时代，你在循环里——你读模型的输出，你决定下一步说什么，你判断什么时候够了。你是控制回路的一部分。

在 loop engineering 时代，你设计循环本身。循环里的每一步可能仍然调用模型（可能很多次），但决定每一步做什么的逻辑是你写的代码、你定的规则、你连的工具——而不是你在聊天窗口里的实时判断。

一个具体例子

假设你要为一个代码库添加一组单元测试。

没有 loop 的做法：你打开聊天窗口，贴一个源文件，说"为这个文件写单元测试"。模型回复一段代码。你复制到项目里，运行测试，发现有失败的，把错误贴回去让模型修。重复三四轮，差不多了。

有 loop 的做法：你写一个脚本，大致是这样——

for 每个需要测试的源文件:
1. agent 读取文件 + 项目中已有的测试风格
2. agent 生成测试代码
3. 运行测试
4. 如果有失败:
a. agent 阅读错误信息
b. agent 修改测试代码
c. 回到第 3 步
5. 如果全部通过 -> commit
6. 如果重试超过 N 次 -> 标记为需要人工审查

你运行这个脚本。它处理了 20 个文件。其中有 17 个一次性通过，3 个经过一两次迭代后通过，1 个标记为人工审查。你花 5 分钟看了那个审查标记的文件，手动调了一行断言。

在这整个过程里，你和模型的对话次数是零。模型被你的 loop 调用了上百次——但你一次都没有打开聊天窗口。

这个定义的两个边界

第一，不是所有任务都适合 loop。一次性的、定义清晰的、不需要迭代的任务——比如"把这个 JSON 转成 YAML"——写 loop 的成本高于收益。Loop 的价值在任务量大、有迭代模式、失败模式可预测的场景里最明显。

第二，Loop 不替代 prompt engineering。你设计 loop 时仍然需要写 prompt——但这些 prompt 现在不是写给最终用户看的人机对话 prompt，而是写给 loop 内部的子代理 的系统指令。它们更结构化，更少自由发挥，更多约束条件。这个区别很重要：你并没有停止写 prompt，你写的 prompt 的消费者变了。

四层抽象阶梯：从对话到自我改进

如果给 2022–2026 年的 AI 工程画一条抽象层次上升的路径，它大致经过四个阶段。每一层包裹但不替换前一层。

第一层：Prompt Engineering（2022–2024）

核心关注点：怎么说话模型才会照做。

你优化的是措辞、角色设定、示例格式、few-shot 排列顺序。你的输出质量取决于你多了解模型的"脾气"——GPT-4 喜欢什么结构，Claude 对什么措辞敏感。这是一门手艺活，而且不透明。同一个 prompt 在两个版本之间可能突然不好用了。

第二层：Context Engineering（2025 年前后）

核心关注点：模型看到了什么。

你不再只关心你说什么，你还关心模型看到的历史、文档、代码库结构。RAG 检索、系统提示组装、对话历史压缩——这些成为核心技能。你的 prompt 不再孤立存在，它被嵌入到一个更大的信息上下文中。

这一层的出现是因为模型开始支持更长的上下文窗口，但很快人们发现"能放进去"不等于"能注意到"。你需要的不是把更多文本塞进窗口——你需要把正确的文本放在模型会读到的地方。

第三层：Harness Engineering（2026 年初）

核心关注点：模型能做什么、不能做什么、怎么约束它。

Harness 是模型和外部世界之间的运行时中间层。它管理工具注册、任务状态、资源预算、可观测性。它决定模型能调用哪些工具、每次调用的超时是多少、连续失败几次应该熔断。

这一层的核心洞察是：模型的可靠性不是模型的问题，是 harness 的问题。给同一个模型配上不同的 harness，产出质量可以差一个数量级。

第四层：Loop Engineering（2026 年中至今）

核心关注点：怎么让系统自己决定下一步。

Loop 是 harness 之上的编排层。它不关心单次调用的质量——它关心的是：系统怎么知道什么时候该做什么？怎么知道自己做完了？做不完的时候怎么办？怎么从一次运行中学到的经验改进下一次运行？

这是最抽象的一层，也是杠杆最高的一层。一个设计良好的 loop 可以把一次手动的三轮迭代压缩成一个自动步骤。但设计它的难度也最高——因为你不再有实时反馈。你设计的时候不知道具体会遇到什么情况。你的 loop 需要在运行时自己判断。

四层的关系

拿我前面那个单元测试的例子来说：

每一轮调用的 prompt 内容是 Prompt Engineering。
让 agent 读取项目中已有的测试风格文件是 Context Engineering。
限制每次调用的 token 预算、超时、重试次数是 Harness Engineering。
决定"对每个文件做同样的事，但失败时循环回来"是整个 Loop。

你不需要精通所有四层来使用 loop——但你设计 loop 越深入，四层都会碰到。这是个全栈活计。

六个原语：Loop 的建筑模块

在研究了多个开源实现和商业产品之后，我发现大多数 loop 可以分解为六种原语。它们不是理论分类——我在实际代码里看到了同样的结构反复出现。

1. 心跳（Heartbeat）——谁启动 loop

Loop 不能凭空运行。它需要一个触发条件。

最简单的是时间触发——cron 表达式，每隔一段时间启动一次。GitHub Actions 的 schedule 事件就是这个模式。更复杂的是事件驱动——webhook 收到外部信号时启动，或者文件系统变化时启动。

Claude Code 内置了 /loop 和 /goal 命令，它们本质上是一个带有交互式心跳的循环：你告诉它一个目标，它在内部决定什么时候该启动下一步。OpenCode 不做这个——它把心跳决策留给你，用 opencode run 作为 worker，你自己用 cron 或 CI 触发器来调度。

心跳是容易被低估的原语。很多人可能会写了一个很好的 loop，但忘了想"这个 loop 是天天跑还是只跑一次"——然后它就只跑了一次，再也没动过。

2. 工作分离（Worktree）——并行隔离

真正的项目从来不是一个 loop 处理一件事。你会有多个 loop 同时在跑——一个修 bug，一个加功能，一个做代码审查。

每个 loop 需要自己的隔离环境。用 git branch 或者独立目录来隔离，这样不同 loop 的修改不会打架。这在实施上很琐碎，但在设计上很关键——没有隔离的 loop 不是并行，是竞态条件。

3. 技能（Skill）——知识固化

Loop 跑一次的时候，它是热的——上下文里全是当前任务的信息。但 loop 跑完，下一轮启动时，它是冷的。它不记得上一轮学到了什么。

Skill 是解决这个问题的原语。它是项目特定知识的固化文件——编码规范、架构决策记录、常见陷阱列表。Skill 文件在每次 loop 启动时被注入到上下文中，让 loop 从冷启动跳到"有备而来"的状态。

好的 skill 不是静态文档。它是从 loop 的运行经验中提炼出来的——一个 loop 发现一个常见错误模式，这个模式被写进 skill，下一次 loop 就不会再犯。

4. 子代理（Sub-agent）——分工与制衡

一个 loop 内部可以只有一个代理反复调用自己，但实践中更好的模式是分工。

我做代码审查的 loop 里有三个角色：一个代理写代码（maker），第二个分出五个影分身从五个正交的角度审查代码(reviewer)，最后一个代理做对抗性验证（adversarial verifier）。Maker 可以很激进——它尝试各种方案，不介意犯错。Reviewer 很保守——它从五个不同的方向逐行审查 maker 的输出，找逻辑漏洞、边界条件、风格偏离。如果 checker 发现了问题，maker 重做。

Maker 和 checker 可以是同一个模型，但角色分离的 prompt 不一样，这就有用了。关键是检查者不能和制造者是同一个对话上下文——否则模型会对自己太温柔。

Adversarial Verifier 只能看到 Maker 的输出，并先入为主地认为有重大缺陷隐藏其中，他的任务就是揪出隐藏的缺陷来。

更复杂的 loop 可以有更多的子代理角色：规划者、执行者、评审者、记录者。子代理越多，协调成本越高，但分工带来更专业的输出。

5. 连接器（Connector）——与现实世界交互

Loop 不能只在模型内部的思维空间里运行。它需要调用工具：读文件、写文件、查数据库、调 API、发消息。

连接器原语处理的是工具如何暴露给 loop 内的代理。MCP（Model Context Protocol）是 2026 年最主流的标准化方案——它定义了一个统一的接口，loop 可以通过这个接口发现和调用外部工具。虽然我颇为嫌弃 MCP 对上下文污染太多，平常很少用到。

连接器的设计决策影响深远：你允许 loop 访问 shell 吗？允许它写任意文件吗？允许它调用生产环境的 API 吗？每个连接都是权限的边界。

6. 脊骨（Spine）——跨运行持久化

这是最被忽视的原语，也可能是最重要的。

模型在每次调用之间遗忘一切。Loop 在每次运行之间也遗忘一切——除非你有地方存状态。Spine 就是这个"地方"。它可以是一个进度文件（progress.md）、一个数据库表、一个看板工具（Linear、Jira）——只要 loop 在下一次启动时可以回答"上次我做到哪了"。

没有 spine 的 loop 不是一个循环——它是一个每次都从第一步开始的重复。

我感觉在 loop 设计里，这个原语的缺席会导致了最隐蔽的失败模式：loop 在跑，也在产出，但产出质量不随时间增长，因为每次启动都把之前的经验丢掉了。

三个最硬的问题

Loop Engineering 目前有三个问题我还没有看到优雅的通用解法。似乎每个项目都得用自己的方式硬扛。

上下文管理

随着 loop 运行轮次增加，模型看到的内容越来越多。但"更多"不等于"更好"——注意力随长度衰减，无关细节淹没关键信号。

通用的应对策略有三类，各有利弊：

压缩：把历史记录总结成更短的形式。丢失细节，但保留主线。
剪枝：直接丢弃不相关的历史。需要启发式判断什么是不相关的——启发式本身可能出错。
外部化：不让模型自己记忆，而是把状态写到一个外部文件里，模型只在需要时读取。最干净的做法，但要额外设计读写机制。

我目前想到的做法是混合使用：每一轮结束时自动做一次压缩（把本轮的关键决策和输出写进 spine），然后在下一轮开始时只加载压缩后的摘要，不加载完整历史。

终止检测

这个问题比看起来难十倍。

一个 loop 需要知道什么时候停下来。听起来很简单——任务完成了就停。但"完成"在复杂任务中是一个很难自动判断的属性。

常见的终止条件有六种：

模型输出了一个明确的终止信号——比如"任务完成"标记。
外部验证通过——测试全绿、lint 无报错、编译通过。
达到最大迭代次数——硬上限，防止无限循环。
超时——时间预算耗尽。
遇到不可恢复的错误——比如依赖服务挂了。
振荡检测——代理反复调用同一个工具、传同样的参数、得到同样的结果。

第六种我花了一段时间才意识到它的存在是必要的。Agent 在遇到困难时会"卡住"——它反复尝试同一个已经证明无效的方案，每次都期待不同的结果。没有振荡检测的 loop 会在这种状态下消耗大量预算。

我对终止的当前判断是：终止不是事后考虑，它是半个设计。一个 loop 的终止条件和它的业务逻辑一样重要。如果你写 loop 的时候先写"做什么"再补"什么时候停"，你得到的一定是一个在边缘情况下会无限跑的 loop。

验证信号

Loop 需要知道自己做的事情对不对。验证信号就是这个"对不对"的输入。

最好的验证信号是确定性的：测试通过、类型检查通过、编译成功。这些信号是 gold standard——它们客观、可重复、不会说谎。

但当任务没有确定性验证可用时（比如写文档、做设计决策），可以退而求其次用模型作为评审者（LLM-as-Judge）。这是一个有用的技术，但它有一个结构性问题：写代码的模型评价自己写的代码太温柔了。不是说它会作弊——而是它会在看到自己熟悉的模式时放松标准。

解决方法我在 maker–checker 前面提到了：制造者和检查者必须是不同的代理实例——最好是不同的上下文、不同的 prompt 体系、甚至不同的模型。这不是不信任——这是架构约束。

两个陷阱：效率的暗面

Loop 越强大，越容易滑入两个陷阱。我说"陷阱"是因为它们不是技术问题——它们是你自己的认知在习惯 loop 之后产生的盲区。

陷阱一：检查仍然是你的工作

Loop 让产出变得更快、更自主。但"完成"是代理的主张，不是证明。一个 loop 说"所有测试通过"——你真的看过测试覆盖了正确的东西吗？一个 loop 说"重构完成"——你真的确认过架构约束没有被绕过吗？

这不是说你不该信任 loop。我信任我的 loop 去做它们被设计来做的事情。但当 loop 的输出要进入生产环境时，审查仍然是你的工作。和人类同事写的代码一样——你信任他们，但你还是做 code review。Loop 生成的代码也是同样的逻辑。

区别在于：因为 loop 产出太流畅了，你很容易跳过审查这一环。流畅性本身是一个认知陷阱。

陷阱二：不要停止深度思考

这是更隐蔽的一个。

Loop 越快地产出你没有亲手写的代码，你的项目能力和你的实际理解之间的差距就越大。你看着一个 loop 重构了整个模块——但你真的能解释重构后的每一行在做什么吗？如果不能，你是模块的拥有者还是旁观者？

我在自己身上看到过这个现象 —— 开始依赖 ClaudeCode/Codex 后，我对代码库的细节掌握明显变薄弱了。不是因为我变懒了 —— 而是因为 AI 代理处理得太好，我没有"被迫理解"的机会了。

这不是反对使用 loop。一个人熟练的工程师用 loop 可以产出比手动高十倍的效率。但那是因为在 loop 出现之前，他已经理解了。他设计 loop 时知道 loop 在做什么，知道边界在哪里，知道什么情况下 loop 的假设会失效。

这个陷阱真正的风险是时间差：如果你在还没有深入理解一个领域的时候就开始用 loop 替你工作，你永远不会有那个"被迫理解"的阶段。loop 把学习的窗户关掉了。

我目前的应对是：对于我熟悉的领域，loop 全速前进。对于我不熟悉的领域，我先手动做几次，理解之后再写 loop。

两条实现路径：工具选择与工程哲学

2026 年中，想实践 Loop Engineering 的人面对两个主流选择：Claude Code 和 OpenCode。它们都做 loop——但背后的哲学差异很大。

维度	Claude Code	OpenCode
心跳机制	内置——`/loop`、`/goal`、Cloud Routines	不内置——用 cron、GitHub Actions 自行调度
工作模式	开箱即用的 loop 原语	`opencode run` 是一个 worker，loop 你自己写
上手门槛	低——内置编排，几分钟能跑	高——需要理解每个零件的原理
厂商锁定	部分存在——Cloud Routines 跑在 Anthropic 服务器上	零——全部本地可控
动态工作流	内置模版和路由逻辑	你的 shell 脚本就是工作流
透明性	封装好的抽象——好用但难改	裸金属——难用但全可控

这个对比的背后是一个更深的选择：你想要一辆造好的车，还是一个发动机让你自己造车？

Claude Code 适合那些想快速体验 loop engineering、不想在基础设施上花时间的团队。它的 loop 原语是内置的，心跳开箱即用，子代理分工通过 /loop 和 /goal 就能实现。你付出的代价是：一旦内置的 loop 模式不满足你的需求，修改起来很困难。

OpenCode 适合那些需要完全控制 loop 行为的团队。你就是 loop 的设计者——心跳自己写，spine 自己选，子代理的调度自己实现。代价是：你要把六个原语全部手动组装一遍。上手慢，但上限高。

我目前倾向 OpenCode 路线。不是因为它更好——而是因为 loop engineering 还在快速演化，我不想把我的 loop 架构绑定在一个厂商的特定实现上。等模式稳定了，再考虑更高的抽象层。

边界条件

本文讨论的 loop 模式在以下情况下可能不适用或需要大幅调整：

单次任务：如果你只需要让模型做一件事一次，loop 是过度设计。写一个 prompt 就够了。
高精度要求：目前 loop 的验证机制在需要人类级判断的任务（法律文件审核、医疗建议）上不够可靠。确定性验证覆盖不到的地方，loop 的可靠性取决于你愿意接受多少错误率。
极低延迟要求：Loop 涉及多轮模型调用，每次调用几百毫秒到几秒。对实时交互有要求的场景，loop 不是一个合适的选择。
团队规模太小：一个人的团队写 loop 可能得不偿失 —— 维护 loop 本身有成本。当团队有三个以上工程师复用同一个 loop 时，这个成本开始物有所值。当然我准备在自己的小项目上实验下 loop，就顾不得这个了。
语言或文化障碍：Loop Engineering 的当前最佳实践和工具文档以英文为主。非母语团队的采纳曲线会更陡。
组织尚未准备好：如果团队的工程文化不支持自动化、不信任自主系统、或者管理方式要求每步都有人签字，loop 的收益会被组织摩擦吃掉。

开放问题

Loop 的可观测性：当一个 loop 跑了 48 小时，调用了上千次模型，做了几百次文件修改——你怎么知道它做得对不对？目前的可观测性工具（trace、log、metric）是为确定性系统设计的，不是为自主决策系统设计的。这个缺口怎么补？
验证泛化：确定性验证（测试、lint）覆盖不了的任务中，LLM-as-Judge 的可靠性到底有多高？我还没有看到严格的、跨场景的基准测试。
跨 loop 状态共享：多个 loop 同时在同一个项目上运行时，它们怎么共享上下文而不会冲突？目前的做法是各自写各自的任务上下文，但"总览全局"的能力在 loop 之间是缺失的。
Skill 的自改进周期：当 loop 从运行中学习到新经验时，它怎么自动更新自己的 skill 文件？这个元循环（改进 loop 的 loop）的实现目前还没有好的通用模式。这个感觉跟 Hermes Agent 有点像。
收敛性保证：多代理 loop（规划→执行→评审→记录）在复杂长周期任务中，是否存在收敛性？有没有一种情况是执行者和评审者反复来回修改同一段代码永远无法达成一致？理论上是的，实际中我见过。什么条件下它一定会收敛？

[[Q]] 一年后重读这篇文章时，我会仍然认同"Loop Engineering 是一次范式跃迁"这个框架，还是会觉得它只是 Harness Engineering 的一个子集？

参考文献

Addy Osmani, “Loop Engineering”, 2026 年 6 月。六个原语的首次系统化定义。
Boris Cherny（Claude Code 创作者），关于"我不再写 prompt，我写 loop"的访谈，2026 年 5 月。
Sydney Runkle（LangChain），“The Art of Loop Engineering”，2026 年 6 月。四层循环栈模型的来源。
Richmond Alake（Oracle），“Agent Loop Decoded”，2026 年 6 月。Agent = Model + Harness 公式和三级成熟度模型。
Peter Steinberger（OpenClaw），关于"技能已转移到 loop 设计"的帖文，2026 年 6 月。

寻找复数网络的并行扫描 —— 从 Mamba 的训练哲学到复数序列模型的高效训练

Sun, 14 Jun 2026 00:00:00 +0000

在一遍遍阅读 Mamba 论文之后，我第一次清晰地意识到：Mamba 训练成功的核心不在于它的选择机制，也不在于它的门控架构。而在于一个更底层、也更数学的事实——它找到了一种将整条序列一次喂入模型、并行算出所有时间步的输出、然后统一反向传播更新参数的方法。

这个能力让它和 Transformer 站在一起，和 RNN 分道扬镳。

而我正在探索的复数神经网络，如果不能在这一点上有所突破，就永远会陷入 RNN 的命运：理论上优美，实践上低效。这篇文章是我对这个问题的当前思考的记录。

核心论点：复数线性递推的数学结合律与实数情况完全一致，因此Mamba的并行关联扫描算法在复数域仍然成立；主要障碍不在数学，而在硬件生态——缺乏针对复数的高效融合CUDA算子。

Scope: 本文讨论从 Mamba 训练方式中提炼的"并行扫描"方法论能否迁移到复数神经网络。覆盖数学论证和工程挑战。不讨论特定复数网络架构（如复数 RNN、复数 LSTM 的具体设计）的细节，也不提供完整实现。

Prerequisites: 假定你了解 Mamba/SSM 的基本架构方向，理解 PyTorch 自动求导的基本机制，并熟悉复数的基础运算性质。

一个问题，两种命运

我花了很长时间才真正理解为什么 RNN 和 Mamba 在数学上都是递推，训练效率却有天壤之别。

RNN 的训练困局可以用一句话概括：每前向传播一步，反向传播就必须退一步。对于长度为 $L$ 的序列，这意味着 $O(L)$ 的串行深度。梯度要沿着时间轴倒着走回序列起点，每一个隐藏状态 $h_t$ 都必须被保存下来供反向传播使用。结果是 $O(L \cdot d^2)$ 的内存和无法并行的计算图。

Mamba 的训练方式完全不同。它走完 $L$ 步前向传播，算出总损失，然后统一做一次反向传播。乍看起来这好像和 RNN 没什么区别——不都是走完再更新吗？但关键在于 Mamba 不走那 $L$ 步串行循环。它用并行关联扫描在一棵深度 $O(\log L)$ 的树里一次性算出所有 $h_1, \dots, h_L$，然后用重计算技术在反向传播时现场重新生成这些状态，避免将它们写入全局显存。

我读到这个机制的时候，心里冒出一个问题：复数版本呢？

复数神经网络的价值在理论上已经被反复论证——复数表示的欠阻尼动态系统有更丰富的表达力、复数递推对旋转和振荡建模更自然、复数梯度流具有与实数不同的特性。但所有这一切，都被训练效率的阴影笼罩着。如果复数 RNN 的训练还是只能 $O(L)$ 串行，它在实际应用中的天花板就是一层天花板。

但等一下。如果 Mamba 的并行扫描依赖的是递推方程的线性性质和由此带来的结合律，而不是递推系数的实数性，那么复数递推方程满足同样的性质。这意味着——至少在数学上——并行扫描对于复数网络是直接可用的。

并行扫描的数学本质

Mamba 能并行的根本原因不在硬件优化，不在 CUDA 编程，而在一个极其朴素的数学性质：结合律。

我们来写出 Mamba 的递推方程：

$$h_t = \bar{A}t h{t-1} + \bar{B}_t x_t$$

这个方程里，$\bar{A}_t$ 是一个 $N \times N$ 矩阵，$\bar{B}_t x_t$ 是一个 $N$ 维向量。我们可以把"从 $t-1$ 到 $t$ 的一步转移"定义为一个操作符 $O_t = (\bar{A}_t, \bar{B}t x_t)$。这个操作符作用于 $h{t-1}$ 的方式是：

$$O_t(h_{t-1}) = \bar{A}t h{t-1} + \bar{B}_t x_t$$

关键来了：相邻两步可以合并。将 $h_t = O_t(h_{t-1})$ 代入 $h_{t+1} = O_{t+1}(h_t)$，得到：

$$h_{t+1} = (\bar{A}{t+1} \bar{A}t) h{t-1} + (\bar{A}{t+1} \bar{B}t x_t + \bar{B}{t+1} x_{t+1})$$

注意结果保持了完全相同的数学形式——一个矩阵乘前一个状态，再加一个向量。因此我们可以定义合并操作 $\otimes$：

$$O_{t+1} \otimes O_t = (\bar{A}{t+1} \bar{A}t, ; \bar{A}{t+1} \bar{B}t x_t + \bar{B}{t+1} x{t+1})$$

这个 $\otimes$ 满足结合律：$(O_3 \otimes O_2) \otimes O_1 = O_3 \otimes (O_2 \otimes O_1)$。这意味着我们可以用树状归约一次性合并所有时间步的操作符，然后用 $O(\log L)$ 深度的并行计算出所有 $h_t$。

这个推导自始至终没有用到任何实数特有的性质。

如果 $\bar{A}_t$ 和 $\bar{B}_t$ 是复数矩阵，$h_t$ 是复数向量，$x_t$ 可能是复数输入，那么：

复数矩阵乘法 $\bar{A}_{t+1} \bar{A}_t$ 满足结合律
复数向量加法 $\bar{A}{t+1} \bar{B}t x_t + \bar{B}{t+1} x{t+1}$ 结合律也成立
整个 $\otimes$ 操作符的推导一行都不需要改

我反复确认了几遍，结论是一样的：在数学层面，复数线性递推的并行扫描条件完全满足。

一个重要细节：选择性机制的复数版本

Mamba 的选择性机制让 $\bar{A}_t$ 和 $\bar{B}_t$ 成为输入 $x_t$ 的函数。在实数域，这通过线性层 $s_B(x_t) = \text{Linear}_N(x_t)$ 等来实现。

对于复数网络，我目前认为有两种方案：

方案一：复数值线性层。将标准线性层的权重和偏置替换为复数版本，直接输出复数值的 $\bar{B}_t$ 和 $\bar{C}_t$。这样做最直接，但需要实现复数版本的线性层及其反向传播。

方案二：实数映射保持。参数保持为实数，通过将实数参数重新解释为复数的实部和虚部来构造复数参数，或者用 $2N$ 维的实数向量表示 $N$ 维复数向量。这种方式不需要修改现有框架的复数支持，但理论上丧失了复数表示的一些结构优势。

我倾向于方案一，因为复数线性层的实现本身并不复杂（只是一个复数矩阵乘法），而且它保持了复数网络的全部表达力。但方案二的一个实际优点是：它可以直接复用现有的实数优化器和 CUDA 库。

这里的 $\Delta_t$ 离散化步长处理更微妙。Mamba 中 $\Delta_t$ 是一个正实数（通过 $\text{softplus}$ 确保正性），经过 ZOH 离散化：

$$\bar{A}_t = \exp(\Delta_t A)$$

如果 $A$ 是复数矩阵，$\exp(\Delta_t A)$ 在数学上定义良好（矩阵指数），但在底层实现上需要复数矩阵指数运算，这比实数版本贵不少。这不是一个原则性障碍——复数矩阵指数已经被广泛研究和实现——但它意味着在实际训练中不能直接照搬 Mamba 的 CUDA 核。

硬件落地的挑战

数学上没有问题，落地就完全不是一回事了。我目前能看到三个层面的障碍，按困难程度排序。

第一层：CUDA 内核支持。 Mamba 的并行扫描之所以快，是因为它有一个定制的融合 CUDA 核（fused kernel），将离散化、并行扫描、输出投影合并为一个操作，全部在 SRAM 中完成，避免 $(B, L, D, N)$ 的中间张量写入 HBM。

对于复数版本，我需要的基础操作（复数矩阵乘法、复数向量加法）本身在 CUDA 中都是支持的——cuBLAS 的 cgemm 和 zgemm 从第一代 CUDA 就存在。问题在于融合。Mamba 的核不是简单的 Scan，而是将多个复数域操作串联成一个融合操作。现有的 cuBLAS 函数做不到这种自定义的融合，我需要自己写 CUDA 核，以及对应的反向传播核和重计算逻辑。

我不确定这样做的开发成本有多高。但有一个折中方案：退回到使用 PyTorch 的 associative_scan 或类似的高层接口。torch.vmap 和 torch.scan 可能提供一部分支持，但效率一定比不上定制核。对于原型验证（proof of concept），这已经够了。

第二层：离散化的复数处理。 Mamba 使用 ZOH（零阶保持）离散化：

$$\bar{A}_t = \exp(\Delta_t A), \quad \bar{B}_t = (\Delta_t A)^{-1}(\exp(\Delta_t A) - I) \cdot \Delta_t B_t$$

当 $A$ 是复数矩阵时，$\exp(\Delta_t A)$ 的计算代价高于实数版本。一个常用的简化是使用欧拉离散化 $\bar{A}_t = I + \Delta_t A$，这在 $|\Delta_t A| \ll 1$ 时效果不错。对于复数网络，这个近似是否成立取决于特征值 $\lambda(A)$ 的模长和 $\Delta_t$ 的取值范围。

另一种思路是使用双线性变换（Tustin’s method），它在复数域同样有效，且数值稳定性优于欧拉法。

第三层：复数的反向传播。 PyTorch 对复数自动求导的支持已经相当成熟（从 1.9 版本开始），支持 torch.complex 和 torch.complex128 的梯度计算。但我没有测试过它对复杂计算图（如并行扫描的重计算模式）的处理效率。自定义反向传播函数（torch.autograd.Function）可能是必要的。

总结一下硬件层面的判断：prototype 可行，性能优化的门槛不低，但不存在原则性障碍。

值得借鉴的现有工作

在做这个调研之前，我以为复数序列模型的高效训练是一个冷门方向。查了一圈发现情况比想象中好。

最直接相关的是一系列 **复数状态空间模型 Complex-valued SSM ** 的工作。Mamba-3 和最新的一些 SSM 变体已经显式地使用了复数值状态。Ran-Milo & Cohen (NeurIPS 2024) 更从理论上严格证明了复数参数化对 SSM 的表达优势：复数 SSM 在中等维度下就能表达实数 SSM 的所有映射，但反之实数 SSM 需要维度达到 $O(t)$（$t$ 为时间长度）；即使维度够高，实数 SSM 的参数值也可能需要指数级大才能表达某些振荡映射，使得它们无法在实际中学习。这为复数 SSM 的研究提供了坚实的理论基础。Mamba 论文本身提到：对于连续模态（音频、视频），复数状态比实数表现更好；对于离散模态（文本、DNA），实数反而更优。这说明复数 SSM 在特定场景下是有明确需求的。

这些工作中的 CUDA 实现主要集中在解决复数矩阵指数和复数并行扫描的数值稳定性问题。我没有找到直接对标 Mamba 融合核的复数版本，但有一些零散的 CUDA 部件可以借用。

另一个方向更间接但同样重要：复数 RNN 的训练加速。虽然传统复数 RNN 没有 Mamba 的并行扫描能力，但最近的工作尝试用梯度裁剪、正交初始化、复数值动量等方法缓解复数 RNN 的训练困难。这些工作虽然不改变串行训练的本质，但提供了一些在无法彻底并行化时的实用改进。比如对复数 RNN 使用基于 Wirtinger 微积分的梯度下降，其收敛性质比简单的实数梯度拆分要好。

还有一个值得关注的线索是复数 FFN 和复数卷积的高效实现。在一些复数图像分类的工作中，“复数计算太慢"的问题通过 cuFFT（快速傅里叶变换）的复用得到了部分缓解——复数卷积用 FFT 计算天然比实数卷积更高效，因为 FFT 本身就是复数运算。虽然这和序列模型的并行扫描不是同一个东西，但它说明复数计算在某些场景下反而有硬件优势。也许复数 SSM 在某些配置下也能用 FFT 加速卷积模式，结合并行扫描获得双重效率提升。

Where this breaks

这个论证目前有几个明显的薄弱环节。我在写的时候就意识到这些问题，写下来供未来自己验证或推翻。

复数矩阵指数是昂贵的。 ZOH 离散化中的 $\exp(\Delta_t A)$ 对于复数矩阵的计算代价远高于实数，特别是在状态维度 $N$ 较大且 $\bar{A}_t$ 每步不同（选择机制）的情况下。欧拉近似或双线性变换可能是必要的简化，它们的近似误差需要被量化。
选择性机制和静态结合的兼容性未验证。 我上面的推导假设了给定输入序列后，$\bar{A}_t$ 和 $\bar{B}_t$ 被确定下来，然后并行扫描对这些确定的（可能是复数的）系数进行合并。但选择性机制本身（从 $x_t$ 求 $\bar{A}_t, \bar{B}_t$）是否在复数域同样稳定？复数线性层的训练收敛性质与实数不同，这可能影响整个流水线。
硬件效率的定量估算缺失。 我说"prototype 可行，性能优化有门槛"但给不出数字。一个完整的复数 SSM 层和实数 SSM 层的吞吐量对比，以及其中离散化步骤的开销占比，需要实际跑实验才能知道。
SRAM 容量限制可能会更紧。 复数占用双倍于实数的存储（float32 vs complex），在 SRAM 这个稀缺资源上，这是一个紧约束。Mamba 的融合核成功的关键之一是状态维度 $N$ 和特征维度 $D$ 的乘积能放进 SRAM。复数版本用 complex 会让占用翻倍，这意味着更小的 $N$ 或 $D$，或者更小的块大小。

Open questions

写完之后留下的问题比写之前更多。列在这里，希望六个月后的我能回答其中的一些。

Q1: 复数并行扫描的 CUDA 融合核开发成本到底有多大？能否基于现有的开源 SSM 代码库（如 state-spaces 仓库）修改，还是需要从零编写？

Q2: 对于复数状态 $h_t$ 而言，离散化误差和数值稳定性与实数版本相比如何？特别是当 $\bar{A}_t$ 的特征值靠近单位圆时。

Q3: 在复数域，Mamba 的选择机制是否仍然有效？或者说，额外的复数自由度是否会带来选择性的一种"自然涌现”？

Q4: 是否有某些任务（如音频合成、振荡动力学建模、量子系统模拟）中，复数 SSM 的 O(log L) 并行训练能力能够带来现实可测量的收益？

[[Q]] 六个月后回看：复数并行扫描的数学可行性我已经确认了，但你实际动手验证了吗？第一个基线实验跑出了什么结果？

References

Gu, Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”, 2023.
Gu, Goel, Ré, “Efficiently Modeling Long Sequences with Structured State Spaces (S4)”, ICLR 2022.
Gu et al., “HiPPO: Recurrent Memory with Optimal Polynomial Projections”, NeurIPS 2020.
Blelloch, “Prefix Sums and Their Applications”, 1990.
Gu et al., “Modeling Sequences with Structured State Spaces”, Stanford PhD Dissertation, 2023.
Arjovsky, Shah, Bengio, “Unitary Evolution Recurrent Neural Networks”, ICML 2016.
Wisdom et al., “Full-Capacity Unitary Recurrent Neural Networks”, NeurIPS 2016.
Trabelsi et al., “Deep Complex Networks”, ICLR 2018.
Ran-Milo, Cohen et al., “Provable Benefits of Complex Parameterizations for Structured State Space Models”, NeurIPS 2024.

Two ways to diffuse text: DFlash block diffusion vs DiffusionGemma

Wed, 10 Jun 2026 00:00:00 +0000

Why this exists

I’ve spent the whole day reading papers at the intersection of diffusion models and text generation, and two names kept surfacing in different contexts: DFlash and DiffusionGemma. Both apply diffusion to discrete text tokens, but they solve completely different problems. DFlash uses a tiny block diffusion model as a fast draft generator inside a speculative decoding loop. DiffusionGemma is a full-scale text diffusion model that replaces the autoregressive decoder entirely. I kept getting them confused — which one uses bidirectional attention? Which one generates 256 tokens at once? Which one guarantees lossless output? This article works through the details side by side so I don’t have to reconstruct the comparison from scratch next time.

Autoregressive language models generate one token at a time. That serial dependency makes them memory-bound at small batch sizes — most of the time goes to loading weights, not computing. Diffusion offers a way out: generate many tokens in parallel and refine them iteratively. DFlash and DiffusionGemma are two concrete implementations of this idea, but they occupy opposite ends of the system-design spectrum.

Scope: This covers the mechanism of DFlash’s block diffusion drafter and DiffusionGemma’s text diffusion model, then compares them across architecture, training, inference, and use case. It does not cover non-diffusion speculative decoding methods (EAGLE-3, Medusa) or image/video diffusion models.

Prerequisites: This assumes familiarity with autoregressive language model basics (causal attention, tokens, logits), the speculative decoding pattern (a small draft model proposes tokens that a large target model verifies), and the general diffusion concept (reverse process that iteratively removes noise).

The bottleneck that diffusion addresses

Autoregressive decoding has a fundamental throughput problem. For each token, the model must load its full set of weights from memory into compute units, do a forward pass, and output a single logit vector. At small batch sizes — typical for interactive applications — the memory-bandwidth bottleneck dominates: the compute units sit idle waiting for weights to arrive. Batching many requests together amortizes this cost, but it does nothing for single-request latency.

Diffusion models flip this dynamic. Instead of one token at a time, they generate an entire block of tokens in parallel during each denoising step. The computation shifts from memory-bandwidth-bound to compute-bound. The question is how to make this work for text, where tokens are discrete and the sequential structure of language matters.

Two distinct answers have emerged. One treats diffusion as a fast approximator inside an existing autoregressive system. The other treats diffusion as the primary generation mechanism, replacing the autoregressive decoder entirely. DFlash is the first approach. DiffusionGemma is the second.

DFlash: block diffusion as a speculative drafter

DFlash was developed at UC San Diego and published at ICML 2026. Its core insight is simple: the hidden states of a large autoregressive language model already encode information about multiple future tokens. Rather than training a separate draft model to predict tokens from scratch, DFlash extracts these hidden features and uses them to condition a tiny block diffusion model.

Architecture

The draft model is a shallow bidirectional Transformer — typically 5 layers — that operates on blocks of $\gamma = 16$ tokens. The generation process works in five steps:

Feature extraction. During the target model’s prefill pass, DFlash extracts hidden representations from a fixed set of layers uniformly sampled from shallow to deep (e.g., layers 3, 10, 17, 24, 31 of a 32-layer model). These hidden states capture information at different levels of abstraction.
Feature fusion and injection. The extracted features are concatenated and passed through a lightweight projection layer that fuses cross-layer information into a compact target context feature. This feature is then injected directly into the Key and Value projections of every draft model layer. The result is stored in the draft model’s KV cache and persists across drafting iterations.

This persistent per-layer injection is the key design choice. Prior work like EAGLE-3 fuses target features only at the input layer, letting the signal dilute as the draft model deepens. DFlash’s approach keeps the conditioning strong regardless of draft depth.
Parallel diffusion drafting. Starting from $\gamma$ random token embeddings, the draft model performs 1-3 bidirectional denoising steps. Unlike autoregressive drafters that must run $\gamma$ sequential forward passes, the diffusion drafter generates all $\gamma$ tokens in a single forward pass per step. Draft latency is nearly independent of $\gamma$.
Parallel verification. The target model processes all $\gamma$ draft tokens in a single forward pass, computing acceptance probabilities for each prefix position. Because the attention computation is parallel within a single sequence, verifying $\gamma$ tokens costs barely more than generating one.
Token acceptance. The longest prefix where all tokens match the target model’s distribution is accepted. One additional “bonus” token is generated for free at the acceptance boundary, and the process repeats from step 1.

Training

The draft model shares the target model’s token embedding and language modeling head — only the draft Transformer layers are trained. Training data comes from response text: random anchor tokens are selected, a contiguous block of $\gamma$ tokens is masked with noise, and the model learns to denoise the entire block in one shot. A position-dependent exponential decay loss weights early tokens more heavily — if the first draft token is wrong, all subsequent tokens in the block are wasted, so the model should prioritize correctness at the beginning of the block.

Multiple blocks can be trained in a single forward pass using sparse attention masks (Flex Attention), with cross-block attention disabled. This makes training efficient despite the block structure.

Why it works

DFlash achieves the best of both worlds. The target model provides high-quality next-token predictions; the block diffusion drafter generates many candidates at near-zero marginal latency. The combination yields lossless 5-6x speedups on models like Qwen3-8B, compared to the 2-3x ceiling of autoregressive drafters like EAGLE-3. On Chain-of-Thought reasoning tasks, where sequences are long and the acceptance distribution matters more, the speedup drops to 4-5x — still far above what AR drafters achieve.

The unconditional baseline (a block diffusion model trained without target features) reaches only 2-3x speedup. The target context conditioning is what makes the difference.

DiffusionGemma: standalone text diffusion

DiffusionGemma is an experimental open model from Google DeepMind, announced in early 2026. It represents a much more ambitious bet: replace the autoregressive decoder entirely with a diffusion process, while building on the Gemma 4 26B A4B backbone.

Noise type: uniform state diffusion

Text diffusion requires defining what “noise” means for discrete tokens. Earlier approaches use masked diffusion, where tokens are replaced with a special [MASK] token. Once a masked position is predicted, it stays fixed — there is no self-correction.

DiffusionGemma uses uniform state diffusion instead. Noise means replacing a token with a random token drawn uniformly from the vocabulary. This has two consequences:

The model must first identify which tokens are noise before it can predict the correct tokens. The denoising task combines detection and correction.
A token predicted in step 1 can be replaced in step 11 if its probability drops. Self-correction is built in.

Rejected tokens (those where the model’s confidence drops) are replaced with fresh random tokens, not the old wrong ones. This keeps the input distribution close to what the model saw during training.

Shared encoder-denoiser architecture

Diffusion models typically need two components: an encoder that processes the user’s query, and a denoiser that cleans the noisy canvas. DiffusionGemma patches a single pre-trained Gemma 4 26B A4B model to serve both roles, rather than training separate networks.

Encoder mode. The model uses its native causal attention to process the user’s query. The resulting KV cache is computed once and stored.

Denoiser mode. Attention is patched from causal to bidirectional, allowing every token in the canvas to attend to every other token. The model uses the logits of all canvas positions to predict replacements. The pre-computed KV cache from the encoder is injected into the denoiser, providing context about the query without requiring cross-attention layers.

This design choice — reusing a single pretrained checkpoint instead of training from scratch — dramatically reduces training cost. The model starts from a capable autoregressive backbone and learns to work in bidirectional mode as a fine-tuning task.

Canvas-based inference

DiffusionGemma operates on a canvas of 256 tokens, generated as follows:

Self-conditioning. The denoiser needs memory of its previous predictions. It takes the probability distribution from the previous step, multiplies it by the embedding matrix (producing one weighted embedding per position), passes it through a small feedforward network, and adds the result as a memory vector to the current step’s token embeddings.
Entropy-bounded sampling. The canvas starts as 256 random uniform tokens. At each denoising step, the model computes entropy for every position. Positions are sorted from lowest entropy (most confident) to highest. Tokens are accepted as long as the cumulative sum of their entropies stays below a threshold. Rejected tokens are replaced with new random values.
Scheduler. A decreasing temperature controls exploration over steps. Early steps use high temperature for broad token search; later steps use low temperature for peaked, confident predictions. Adaptive stopping halts denoising early if the model is stable (top tokens unchanged for N steps) and confident (entropy below 0.005).
Block extension. For text longer than 256 tokens, the generated block is appended to the prompt, the KV cache is cheaply updated (it is causal on the encoder side), and diffusion begins on the next 256-token canvas.

The result is a model that generates up to 4x faster raw token throughput than an autoregressive baseline on dedicated GPUs, though the quality on complex reasoning tasks trails the full Gemma 4 autoregressive model by a small margin.

Side-by-side comparison

With both mechanisms on the table, the differences snap into focus.

Dimension	DFlash block diffusion	DiffusionGemma
Role	Draft model in speculative decoding	Standalone text generation model
Output guarantee	Lossless — target model verifies	None — quality depends on denoising
Model size	5-layer bidirectional Transformer (~tens of millions params)	Full 26B backbone with bidirectional patch
Block size	16 tokens	256 tokens
Denoising steps	1-3	Many (scheduler-controlled, adaptive)
Noise type	Masked diffusion (predict corrupted tokens)	Uniform state diffusion (random tokens)
Self-correction	Not needed (target catches errors)	Yes — tokens can be revised
Attention	Bidirectional within block	Dual: causal (encoder) / bidirectional (denoiser)
Conditioning source	External: target model hidden features	Internal: self-conditioning from previous step
Conditioning injection	Into K/V projections of every draft layer	Added as memory vector to token embeddings
KV cache	Draft model builds its own with injected features	Encoder computes once, denoiser reuses
Extended generation	Speculative decoding loop (draft → verify → accept)	Block diffusion (append canvas → update KV cache → next canvas)
Speedup over AR	5-6x system-level (draft + verify combined)	Up to 4x raw token throughput
Primary bottleneck	Acceptance length (how many draft tokens are accepted)	Denoising step count

Where they converge

Both models exploit the same fundamental idea: bidirectional attention within a block of tokens allows parallel prediction, and the block structure enables integration with autoregressive generation (DFlash through verification, DiffusionGemma through block extension). Both achieve their best speedups when compute, not memory, is the limiting factor — which means dedicated GPUs, not CPU inference or heavily loaded servers.

Where they diverge

The divergence is best understood through the lens of error handling. DFlash accepts that its draft model will make mistakes and handles them through verification — the target model catches errors, so draft quality just affects throughput, not correctness. DiffusionGemma cannot afford to make mistakes because there is no verifier. It must allocate enough denoising steps for the model to self-correct, which creates a direct quality-speed tradeoff.

This difference cascades through every design choice. DFlash can use a tiny draft model because correctness is not its job. DiffusionGemma needs the full 26B backbone because the model must produce acceptable text on its own. DFlash can use 1-3 denoising steps because rough drafts are fine. DiffusionGemma needs many more because each step should improve the output. DFlash uses masked diffusion (simpler, cheaper) because errors are caught downstream. DiffusionGemma uses uniform state diffusion (more flexible, more expensive) because self-correction is essential.

When to use which

DFlash is the right choice when you already have a deployed autoregressive model and want to reduce latency without changing the output distribution. The integration cost is moderate — you need to train the small draft model and set up the speculative decoding pipeline — but the output is guaranteed identical to the original model. It is particularly effective for long generations (chat, reasoning chains) where the acceptance distribution has time to average out.

DiffusionGemma is the right choice when you are building a new system from scratch and want the fastest possible raw generation speed, or when you are experimenting with text diffusion as a research direction. The quality gap relative to autoregressive models is small and narrowing, and the 4x speedup at small batch sizes is genuine. But you are committing to a different output distribution, and there is no verifier to catch mistakes.

The two approaches are not direct competitors. They operate at different levels of the system stack and solve different constraints. DFlash accelerates an existing model. DiffusionGemma replaces one.

Where the comparison breaks

The speedup numbers are not directly comparable. DFlash’s 5-6x includes both draft generation and target verification in an end-to-end system. DiffusionGemma’s 4x is raw token throughput from a single model. If you were to run DiffusionGemma through a speculative decoding loop with a separate verifier, the effective speedup would be different.

The acceptance length metric ($\tau$) that DFlash reports has no direct analogue for DiffusionGemma. A diffusion model does not produce a sequence that an external verifier accepts or rejects — it produces text directly. Comparing $\tau = 6.8$ to a 256-token canvas is comparing apples to power plants.

Both approaches assume you have a GPU with enough memory. DFlash needs memory for both the target model and the draft model (though the draft model is tiny). DiffusionGemma needs memory for the full 26B backbone plus the bidirectional attention workspace. Neither works well on CPU or edge devices.

Open questions

Does the block diffusion draft model benefit from larger block sizes ($\gamma > 16$) in expectation, or does the acceptance distribution saturate? DFlash’s block-size transfer experiment (large blocks train → small blocks infer) suggests a non-trivial relationship.
Can DiffusionGemma’s encoder-denoiser sharing be applied to smaller backbones (4-7B) without unacceptable quality loss? The Gemma 4 26B is a large starting point.
What happens when you stack the two approaches — use a block diffusion model as a draft model for DiffusionGemma as the target? Would the uniform state diffusion’s self-correction interact well with a separate verifier?
How does the quality-speed Pareto frontier of DiffusionGemma shift with fewer denoising steps plus a small external verifier (a hybrid approach)?
The position-dependent loss decay in DFlash is motivated by speculative decoding’s sequential constraint. Does a similar non-uniform loss help DiffusionGemma’s training, even though all positions matter for final output quality?

[[Q]] Six months from now: has any work combined the two approaches — using a block diffusion drafter conditioned on a diffusion target model’s hidden states, or alternatively, has DiffusionGemma’s uniform state diffusion been explored as a training objective for draft models?

References

Chen, Liang, Liu, “DFlash: Block Diffusion for Flash Speculative Decoding”, ICML 2026, arXiv:2602.06036.
Google DeepMind, “DiffusionGemma: An experimental open text diffusion model”, 2026. Blog post and developer guide.
Li et al., “EAGLE-3: Scaling Speculative Decoding via Training Objectives”, 2025.
Samragh et al., “Hidden Features of Large Language Models Encode Future Tokens”, 2025.
Nie et al., “LLaDA: Large Language Diffusion with Masked Language Modeling”, 2025.
Arriola et al., “Block Diffusion Models”, 2025.

Pruning Qwen3.6-35B-A3B for RTX 5090: what I learned pushing MoE compression to its limit on a single GPU

Mon, 18 May 2026 00:00:00 +0000

Why this exists

I had a 35B-parameter MoE model that needed to run on a single RTX 5090. The model in FP8 needed 34.4 GiB. The GPU had 31.84 GiB. The gap was 2.6 GiB — small enough to feel tantalizing, large enough to break every standard deployment pipeline.

Six days later, I had a pruned model that scored 73.2% on HumanEval+, 51.0% on Toolcall, and 33.6% on MMLU. That model (v3) shipped as the best result available at the time.

Twenty-four hours after that, I had evidence that everything I thought I understood about calibration was specific to one recipe and didn’t generalize. Then another twelve hours produced v7b-fp8 — a model that beats v3 on every measured pack I ran, with BugFind +17, DataExtract +17, and InstructFollow +20.

The path between those states was not linear. Some of my most confident conclusions turned out to be wrong. This article is what I wish I had known on day one — updated after the sessions that overturned my earlier priors.

Qwen3.6-35B-A3B is a 256-expert MoE model optimized for agentic coding, and my only available hardware was a single RTX 5090. This covers pruning, quantization, evaluation, and attempted recovery fine-tuning on a single consumer GPU across seven experimental sessions. It does not cover multi-GPU setups, cloud inference, or comparisons with other pruning algorithms.

Why a 2.6 GiB deficit ate my week

The standard view is that compressing a model by 8% to fit a memory budget is a routine calibration exercise — run AWQ or GPTQ, adjust the quantization config, ship it. A 2.6 GiB gap on a 34.4 GiB model is 7.5% compression. Well within what quantization alone should handle.

It was not routine.

The contradiction hit immediately: BitsAndBytes, GPTQ, and AWQ all quantize only 2D nn.Linear weights. MoE models store experts as batched 3D tensors — [n_experts, in_dim, out_dim] — for efficient grouped matrix multiplication. These 3D tensors contain roughly 90% of the model’s total parameters, and every standard quantization tool simply skips them. No error. No warning. They just pass through at full precision.

So quantization alone could not close the gap. I needed expert pruning.

REAP (Router-Weighted Expert Activation Pruning) scores each expert by the conditional average of its gating weight times its activation norm over calibration tokens:

$$S_j = \mathbb{E}_{x \in X_j}[g_j(x) \cdot |f_j(x)|]$$

The intuition: an expert that gets low gating weight and produces small activations contributes little to the output and can be removed with minimal reconstruction error. REAP then removes the lowest-scoring experts and propagates residuals to keep the functional manifold topology intact.

The pruner works block-by-block: load one 750 MiB decoder layer to GPU, score all surviving experts, prune the lowest, propagate residuals, save. About 25 minutes per experiment across 40 decoder layers. I ran seven experiments over seven sessions on one GPU.

The constraint that shaped everything: the RTX 5090’s 32 GiB was simultaneously my inference platform, evaluation framework, and training environment. Pruning, eval, and SFT all competed for the same VRAM, and no two of them could run at the same time. At BF16, the model occupied 30.6 GiB — 97% of GPU capacity, leaving zero room for gradients or activations during training.

This total-resource competition, not the pruning algorithm, turned out to be the hard problem.

The calibration mix that rewrote my priors

I came to this expecting the pruning algorithm or the compression ratio to dominate quality. That is how the literature frames it: better importance scores, better pruning decisions.

The evidence says otherwise.

Here is what happened.

My first pruning experiment (v2) used four code-focused calibration datasets: evol-codealpaca, BigCodeBench, SWE-bench, and xlam. Pure code data. The result: HumanEval+ at 72.0%, Toolcall at 44.0%, and MMLU at — two categories at 0.0%.

Dead categories. The model could generate code, but it could not answer a general-knowledge question. The pruner had never seen general-knowledge tokens during scoring, so it had no way to know which experts mattered for those domains. Any expert carrying general-knowledge information was pruned or severely weakened.

For the v3 experiment, I switched to a 70/30 code-to-general mix. Same REAP algorithm. Same compression ratio. Just two general datasets added — 600 samples of MMLU and 600 samples of C4 — to a pool of 700 samples each from four code datasets.

Category	v2 (pure code cal)	v3 (70/30 cal)	Change
MMLU Social Sciences	0.0%	33.3%	+33.3pp
MMLU Other	0.0%	34.3%	+34.3pp
HumanEval+	72.0%	73.2%	+1.2pp
Toolcall	44.0%	51.0%	+7.0pp

Every single metric improved. MMLU recovered from dead to 33%+. Code benchmarks improved too.

I think of calibration data as the lens through which the pruner sees the model. A narrow lens (pure code) gives a sharp but myopic view — the pruner keeps only what it sees, and blinds the model to everything else. A wider lens (70/30 mix) lets the pruner see the full functional space, so it preserves the structures that serve the whole distribution, not just one mode.

The analogy breaks in one direction: you cannot just keep adding calibration domains forever. More data means longer scoring passes. But within a practical budget, the evidence is clear — calibration composition matters more than any algorithmic tweak I could have made to the pruner itself.

I shipped v3 and moved on. That was where the story got interesting.

The discovery that my conclusions were calibration-specific

After shipping v3, I went back to test a hypothesis that seemed obvious: if I replaced the code-heavy calibration with agentic traces, the pruned model would perform better on agentic benchmarks. Tool calling, bug finding, multi-step reasoning — these were what the model was designed for.

I built a proper BenchLocal evaluation harness — 8 packs covering tool-call, hermes-agent, bug-find, data-extract, instruct-follow, reason-math, struct-output, and cli — and established a v3 baseline with the new v19 chat template. The baseline was sobering: ToolCall-15 at 90, HermesAgent-20 at 16, BugFind-15 at 8. The agentic gates were low to begin with.

I ran two candidate experiments at a deeper compression (0.40, keeping 154 of 256 experts) with two calibration strategies:

Mix-A: full replacement — agentic data (glm47-reap + hermes-agent-traces) instead of code
Mix-B: additive — agentic data layered on top of v3’s base

Both produced the same result. ToolCall-15 dropped from 97 (the v3 baseline with the old reasoning parser) to 90. HermesAgent-20 stayed at exactly 16. BugFind-15 hovered in the 3-10 range. No agentic movement. Both failed all three gates.

I stopped and ran a control experiment. I took Mix-A’s exact calibration list and ran it at v3’s exact compression ratio (0.289, 183 experts). If compression depth was the cause of the toolcall regression, the control would recover toolcall. If calibration content was the cause, the control would still show the regression.

The control was decisive:

Candidate	Compression	Calibration	ToolCall-15	HermesAgent-20	BugFind-15
v3	0.289	70/30 balanced	97	16	8
Mix-A	0.40	Agentic replacement	90	16	10
Mix-B	0.40	Additive layering	90	16	3
v3ratio	0.289	Mix-A calibration	90	16	0

ToolCall-15 at 90 across all three candidates, including the control at v3’s compression. The regression came from the calibration content, not the compression depth.

The mechanism was surprisingly specific. The regression was concentrated entirely in a single sub-dimension: Parameter Precision dropped from 100 to 67. The model still picked the right tools with the right structure — it just generated wrong-typed or wrong-formatted arguments more often. Dropping code corpora (evol-codealpaca, bigcodebench, swe-bench) and substituting agentic traces had cost the model its tight argument-formatting discipline.

The other finding was harsher. HermesAgent-20 scored 16 out of 20 across all four configurations — literally identical, including per-category breakdown. A 25B pruned MoE cannot handle these multi-step browser-automation scenarios, regardless of what you feed it during pruning calibration. The gate is capacity-bound.

I also discovered that the vLLM --reasoning-parser qwen3 flag was essential for correct evaluation. Without it, the model’s <think> reasoning block leaked into all plain-text responses, breaking every non-tool-call scorer. The flag lifted ToolCall-15 from 90 to 97 and recovered instruct-follow and data-extract from flat zero. The lesson: validate your eval infrastructure before you trust a single number.

I closed Session 6 with a clean negative result and the conviction that calibration content could not move agentic gates. That conviction lasted about twelve hours.

The recipe that broke through

Session 7 adopted a fundamentally different calibration recipe: the REAP-26B 6-dataset mix. Six datasets — SWE-bench/SWE-smith-trajectories (tool split), xlam-function-calling-60k, evol-codealpaca, and Mixture-of-Thoughts (code/math/science) — at much higher token count (1024 samples x 16384 sequence length, totaling 16.8M tokens). Router renormalization disabled per the REAP-26B README.

I ran three plans plus a follow-up:

Plan-A (compression 0.40, fresh prune). ToolCall-15 collapsed to 63 — a catastrophic -27 regression. But BugFind jumped to +15 and InstructFollow to +33. The recipe was clearly powerful. Too powerful at this depth.

Plan-C (stacked prune on top of the upstream REAP-26B-VL). Recovered toolcall to 90 but lost ~90% of the recipe’s other gains. Stacked pruning does not inherit upstream calibration signal.

Plan-B (compression 0.289, v3’s depth, the experiment I initially skipped). This was the follow-up after both Plan-A and Plan-C failed.

Candidate	Compression	ToolCall-15	BugFind-15	DataExtract-15	InstructFollow-15	Verdict
v3+v19	0.289	90	8	5	20	Baseline
v7a	0.40	63	23	24	53	FailToolcall
v7c	stacked	90	0	4	16	FailAgentic
v7b	0.289	93	25	22	40	Pass

v7b-fp8 scored equal or better than v3 on all 7 measured packs. No regressions. BugFind +17, DataExtract +17, InstructFollow +20, ToolCall +3. The trigger verdict was Pass.

This is the result I keep coming back to: the same recipe at 0.40 collapsed toolcall to 63; at 0.289 it improved toolcall to 93. The recipe drives the agentic gains. The compression depth modulates the toolcall trade-off. Session 6’s conclusion that “calibration content cannot move agentic benchmarks” was specific to Mix-A’s content, not a universal property of pruning calibration.

The pipeline upgrades that made this possible are worth noting. The 16K sequence length calibration required a chunked REAP scoring accumulator — the single-pass approach would have materialized 67 GiB on a 32 GiB GPU. The custom FP8 quantizer (scripts/quantize_fp8.py) bypasses llmcompressor’s broken Qwen3.6 compatibility with a 274-line direct cast from BF16 to torch.float8_e4m3fn. Schema adapters with a 60-second preflight caught dataset drift before any GPU allocation.

How 3D tensors broke every quantization framework

The calibration experiment got me v3 — a model that almost passed its quality gates. HumanEval+ at 73.2% was close to the 75% threshold. MMLU at 33.6% was still short of the 40% target. The obvious next step was recovery fine-tuning.

This is where the second assumption broke.

I assumed that standard quantization tools would handle model compression for training. Load the model in 4-bit, apply LoRA adapters, train. This is the default workflow for QLoRA on every Hugging Face tutorial. It works on LLaMA, it works on Mistral — it should work on Qwen.

It does not. Because the model’s 3D expert tensors are invisible to BitsAndBytes.

I spent the evening of day two systematically eliminating every standard training approach. Seven attempts, all failures:

Attempt	Approach	Result
1	BnB 4-bit QLoRA	Can’t quantize 3D expert tensors [183, 1024, 2048]
2	BF16 model.to(‘cuda’)	30.6 GiB — 0 bytes left for activations
3	accelerate device_map=‘auto’	Keeps all layers on GPU for backward
4	DeepSpeed ZeRO-3 (single GPU)	Trainer moves model to GPU before partitioning
5	DeepSpeed zero.Init + from_pretrained	Weight loading conflicts with meta-device tensors
6	FP8 frozen weights + monkey-patched ops	grouped_mm upcast creates 768 MiB BF16 temp per layer
7	FP8 + dispatch_model with 10 GiB budget	Offloaded layers accumulate on GPU during backward

I don’t fully understand why every framework eventually calls model.to(device) for the backward pass on single GPU. The documentation promises CPU offloading. The reality is that DeepSpeed ZeRO-3, accelerate’s dispatch_model, and FSDP all converge on the same behavior: put the full model on GPU when gradients need to flow.

The resolution came from a workaround I had not considered: unbatch the 3D expert tensors into individual bnb.nn.Linear4bit layers. BnB can quantize standard 2D linear layers. A 3D tensor [183, 1024, 2048] becomes 183 separate Linear(2048, 1024) objects, each quantizable in 4-bit.

The result: model on GPU dropped from 30.6 GiB to 16.8 GiB, leaving 16.9 GiB for activations and gradients. The SFT ran — 311 steps, 9,934 samples, 11.5 hours, loss from 1.058 to 0.975, token accuracy from 85% to 96%. By every training metric, it worked.

It did not work.

The SFT trap: when training makes everything worse

I expected SFT with quantized frozen weights to improve the model. The training curves were healthy. Loss decreasing. Token accuracy climbing. All the signals that normally say “keep training, it is converging.”

Post-SFT evaluation told a different story:

Benchmark	Pre-SFT	Post-SFT	Delta
HumanEval+	73.2%	67.7%	-5.5pp
Toolcall	51.0%	50.5%	-0.5pp
MMLU	33.6%	9.4%	-24.2pp

Everything regressed. MMLU collapsed back to v2-level. HumanEval+ lost 5.5 points.

The mechanism is specific and instructive — and worth pausing on because it explains an entire class of pipeline failures:

4-bit quantization injects noise into the forward pass of every frozen expert layer. That noise is deterministic — same input, same 4-bit weights, same quantization error — but it shifts the activation distribution that the trainable router and shared expert layers see. The trainable parameters adapt to this shifted distribution during SFT. They learn to work with the noise profile of the 4-bit experts.

When you remove the 4-bit quantization (merge the fine-tuned weights back into the original BF16 model for inference), the noise profile disappears. The trainable parameters are now running on clean activations, and they have overfit to a distribution that no longer exists.

This is why the training metrics looked great while the benchmarks collapsed. The model was not learning to generate better code or answer knowledge questions. It was learning to compensate for quantization noise in the frozen pathway. When the noise went away, the compensation became output distortion.

The result that I keep returning to: the pre-SFT v3 model was the project’s best output at that point. The calibration strategy was the lever. Fine-tuning was a trap.

What I would do differently

If I started this project again, I would change several things. The later sessions taught me that some of my early conclusions were incomplete — so these recommendations are updated with everything I know after seven sessions.

What I did	What I would do instead	Why
Jumped straight to production-scale SFT	Validate on a 2-layer toy model first	Seven failed approaches, ~4 hours of debugging, caught in minutes
Let eval and training share one venv	Isolate venvs from the start	huggingface_hub 1.5 vs 1.14 broke vLLM weight loading
Ran SFT before exhausting calibration experiments	Run all calibration experiments before SFT	The 50/50 experiment was never attempted, and calibration is the primary lever
Shallow pruning (183/256) + 4-bit SFT	Deeper pruning (154 experts) + clean BF16 SFT	Avoids the noise-overfitting trap entirely
Assumed agentic calibration lifts agentic gates	Test the REAP-26B recipe at the original depth first	Session 6’s clean negative result was Mix-A-specific; Plan-B at v3’s depth passed everything
Tested one variable at a time	Test “new calibration at same compression” as default isolation	The v3ratio control flipped the interpretation — always isolate the variable
Championed “capacity-bound benchmarks” as a universal conclusion	Measure with multiple recipes before declaring a ceiling	BugFind moved +17 with the right recipe at no parameter count change

The most painful lesson is also the most transferable: validation loops catch pipeline bugs, but they do not catch strategy bugs. The SFT pipeline ran correctly — no crashes, no OOM, healthy training curves — and produced a worse model.

The one experiment I most regret not running is the 50/50 calibration mix. If 30% general data pushed MMLU from 0% to 33%, 50% might push it past 40%. That experiment would have taken 25 minutes. The SFT that replaced it took 11.5 hours and made everything worse.

Boundary conditions

The calibration-composition result is established for REAP on one model family (Qwen3.6-35B-A3B). It likely transfers to other MoE models and other pruning algorithms, but I have not tested this.
The REAP-26B recipe finding (v7b-fp8 beating v3) is specific to this calibration mix at this compression depth. Whether it generalizes to other MoE scales is an open question.
The Session 6 “calibration content drives toolcall regression” finding is specific to Mix-A’s content (glm47-reap + hermes-agent-traces). The REAP-26B recipe at the same depth showed a positive toolcall delta. The finding is recipe-specific, not universal.
HermesAgent-20 remained stuck at 16/20 across all seven sessions and all configurations tested. It is genuinely capacity-bound at this model size.
The 4-bit expert unbatching technique works at the cost of inference speed — per-expert sequential Linear4bit forward passes are slower than native grouped matrix multiplication.
The SFT degradation result applies specifically to training with 4-bit frozen experts on this model architecture. FP8 frozen experts or full BF16 SFT may behave differently.
Single-GPU constraints shaped every conclusion. With multi-GPU hardware, the trade-offs shift substantially.
The v19 chat template costs about 7 toolcall points compared to v18 (90 vs 97 on the same model). All Session 7 comparisons are within the same template, but direct comparability with Session 6 numbers requires accounting for the template shift.

Open questions

Can SFT on v7b-fp8 lift HermesAgent-20? It is the only pack that v7b didn’t improve, stuck at 16/20. Session 6’s capacity-bound conclusion was wrong for BugFind (the right recipe moved it +17). It may also be wrong for HermesAgent once the right recipe is found. But SFT on actual agent traces is a qualitatively different approach, and the infrastructure exists but hasn’t been validated against v7b.
Does 50/50 calibration push MMLU past 40%? The experiment takes 25 minutes and was never scheduled.
Can Transformer Engine FP8 training enable quality SFT without the noise-overfitting trap? The tools are installed on sm_120. Untested.
Does the REAP-26B recipe replicate on other MoE families? The recipe drove +17 across multiple benchmarks on Qwen3.6. Would it produce similar gains on DeepSeek, Mixtral, or OLMoE?
Should stacked pruning ever be used? Session 7 showed it destroys upstream calibration signal. But if the upstream calibration is expensive (24K samples on a 96 GB GPU), stacking a cheap re-prune on top seems like it should work in theory. The empirical result was negative. I don’t fully understand why.

[[Q]] Six months from now: run the 50/50 calibration experiment first. If it pushes MMLU past 40%, the entire SFT effort was wasted. And promote the v7b symlink — it is the best model you have.

References

Fang et al., “REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression”, arXiv:2510.13999, 2025.
Dery et al., “Finding Fantastic Experts in MoE Models”, arXiv:2504.15447, 2025.
Zhang et al., “Efficient Expert Pruning in MoE LLMs”, arXiv:2505.12345, 2025.
BitsAndBytes, Hugging Face quantization library, https://github.com/bitsandbytes-foundation/bitsandbytes.
TRL: Transformer Reinforcement Learning, Hugging Face, https://github.com/huggingface/trl.