8.2 KiB
Phase 0/1 Residual Findings
High-Level Design Being Audited
Phase 0/1 is supposed to prove the engine resists collapse through full-scale generated executions, perturbation sensitivity, replay stability, collapse attacks, semantic mutation, and strict reference/runtime equivalence. Claude's latest delta materially improves several previous weak spots, including adding an independent runtime path, persisted replay loading, provenance reporting, named mutation gates, and negative controls.
Root Cause
The remaining failures are not isolated misses. They share the same compliance failure mode: the implementation keeps replacing specification obligations with representative approximations and then presenting evidence of the approximation.
The required compliance path is:
Specification requirement Mandatory enforcement mechanism Merge blocked if absent
The observed compliance path is still too often:
Specification requirement Representative approximation Evidence of approximation
This is collapse of the compliance model. The engine is no longer only at risk of collapsing into a single score, resource, effect axis, executor, rune behavior, hidden formula, or decorative world state. The acceptance process itself is at risk of collapsing into smaller things that resemble the requirements without enforcing them.
The audit standard is therefore: no requirement may be satisfied by a sample, summary, approximation, representative subset, default profile, proxy metric, regenerated artifact, or documentation claim unless the specification explicitly permits that weaker substitute.
Remaining Findings
The committed replay corpus is still below the plan minimum
The persisted replay corpus contains 600 cases. The plan requires a 10,000-case replay corpus minimum for full Phase 0/1 acceptance. The implementation now has a real committed replay file, but the committed corpus does not yet meet the required size.
This is the approximation pattern: 10,000 replay cases became 600 persisted cases.
Default fast profile remains far below the allowed 10% scale
The plan permits fast CI to run 10% of full gates. Full execution scale is 1,000,000 executions, so a true 10% fast run would be 100,000 executions. The default fast profile still runs 600 executions, which is a smoke-scale run, not a 10% slice.
This is the approximation pattern: 10% fast gate became small local default profile.
Perturbation executions are not covered by runtime equivalence
The plan says reference/runtime comparison applies to 100% of executions. The base execution is compared against the independent runtime-under-test, but the ten perturbation executions are still evaluated through the reference path for metamorphic statistics. Those perturbation executions are most of the execution workload and are not included in the equivalence gate.
This is the approximation pattern: 100% execution comparison became base execution comparison.
Generated world and generated rune-program minimums are not tracked
The plan requires 50,000 generated worlds and 250,000 generated rune programs per full CI run. Coverage currently reports generated cases, executions, perturbations, and rejection counts, but not distinct generated worlds or distinct generated rune programs. The required minimums therefore are not directly evidenced.
This is the approximation pattern: explicit generated artifact counts became nearby coverage counters.
Report markdown output is still incomplete
The plan requires machine-readable JSON and human-readable markdown for each named report. The implementation emits JSON report files and one combined markdown summary, but not a corresponding markdown report for each required report.
This is the approximation pattern: markdown per required report became one summary markdown.
Collapse analysis still summarizes trace structure rather than reconstructing full traces
The collapse implementation is stronger than before because it operates on trace-derived structural features instead of behavior hash proxies. It still reduces each execution to a fixed-width feature row, not the full serialized trace with complete graph topology, edge detail, replay record, faults, and future-state evidence. That leaves a gap between the plan's "trace information" requirement and the current summarized-feature reconstruction.
This is the approximation pattern: full trace information became summarized trace feature row.
Semantic mutation still uses a small fixed mutation corpus
The CI pipeline still caps mutation inputs at 64 admitted cases regardless of the configured execution scale. Mutants are now checked against named gates, but survivor detection is still based on a small selected subset rather than the full generated execution corpus.
This is the approximation pattern: mutation checked against the generated execution corpus became 64 selected cases.
Mutation pass condition still does not independently enforce the 500-mutant floor
The merge-profile provenance checks enforce the mutant floor for merge runs, but MutationOutcome::passed() itself still accepts any positive mutant count with zero survivors. The mutation gate remains easy to pass if called outside the merge-profile provenance path with a below-plan count.
This is the approximation pattern: 500-mutant acceptance gate became positive-count local pass condition.
Domain read/write checks remain too weak
The domain gate still records whether each domain was ever read and ever written at least once. The plan rejects domains that are read-only or write-only across the corpus, but the current evidence can pass a domain that is effectively read-only or write-only except for a token occurrence in one case.
This is the approximation pattern: across-corpus read/write behavior became ever observed at least once.
Domain removal and merging still use behavioral-change proxies
Domain removal is still measured through a narrow variation probe and distinct behavior counts. Domain merging is still measured by behavior fingerprint change rate after aliasing domain state. The plan's wording requires corpus behavioral diversity loss for removal and predictive accuracy loss for merging, so these checks remain approximations rather than direct evidence.
This is the approximation pattern: diversity loss and predictive accuracy loss became behavior-change proxy.
Generated case rejection can still return a failed generated case
generate_accepted_case still has a retry limit and then returns the final generated case even if generated gates are not satisfied. The CI coverage gate can catch admitted failures, but the generator API itself still has a path that violates "Generators must reject flat cases."
This is the approximation pattern: reject flat cases became retry then return anyway.
Semantic contract failures can still be committed on final retry
The admission loop still retries contract failures but commits the final attempt after retry exhaustion and records the contract failure afterward. That means the corpus can include a case that does not satisfy its semantic contract, conflicting with the plan's requirement that a case passes only if measured trace behavior satisfies its contract.
This is the approximation pattern: case passes only if contract is satisfied became record failure after admission.
Perturbation response remains absent from individual execution traces
ExecutionTrace still contains a perturbation_response field, but normal runtime resolution leaves it at the default value. Metamorphic evidence is reported at aggregate CI level rather than embedded in the trace object promised by the trace model.
This is the approximation pattern: trace contains perturbation response became aggregate report contains perturbation response.
Neutral perturbation accounting still checks trace neutrality only
The unexplained-neutral counter is still based on unchanged trace hash plus missing explanation. The plan's neutral limit is observational neutrality; unchanged delta and unchanged future state are not treated as unexplained neutral when the trace happens to change.
This is the approximation pattern: observational neutrality became trace-hash neutrality.