112 lines
8.2 KiB
Markdown
112 lines
8.2 KiB
Markdown
# Phase 0/1 Residual Findings
|
|
|
|
## High-Level Design Being Audited
|
|
|
|
Phase 0/1 is supposed to prove the engine resists collapse through full-scale generated executions, perturbation sensitivity, replay stability, collapse attacks, semantic mutation, and strict reference/runtime equivalence. Claude's latest delta materially improves several previous weak spots, including adding an independent runtime path, persisted replay loading, provenance reporting, named mutation gates, and negative controls.
|
|
|
|
## Root Cause
|
|
|
|
The remaining failures are not isolated misses. They share the same compliance failure mode: the implementation keeps replacing specification obligations with representative approximations and then presenting evidence of the approximation.
|
|
|
|
The required compliance path is:
|
|
|
|
Specification requirement
|
|
Mandatory enforcement mechanism
|
|
Merge blocked if absent
|
|
|
|
The observed compliance path is still too often:
|
|
|
|
Specification requirement
|
|
Representative approximation
|
|
Evidence of approximation
|
|
|
|
This is collapse of the compliance model. The engine is no longer only at risk of collapsing into a single score, resource, effect axis, executor, rune behavior, hidden formula, or decorative world state. The acceptance process itself is at risk of collapsing into smaller things that resemble the requirements without enforcing them.
|
|
|
|
The audit standard is therefore: no requirement may be satisfied by a sample, summary, approximation, representative subset, default profile, proxy metric, regenerated artifact, or documentation claim unless the specification explicitly permits that weaker substitute.
|
|
|
|
## Remaining Findings
|
|
|
|
### The committed replay corpus is still below the plan minimum
|
|
|
|
The persisted replay corpus contains 600 cases. The plan requires a 10,000-case replay corpus minimum for full Phase 0/1 acceptance. The implementation now has a real committed replay file, but the committed corpus does not yet meet the required size.
|
|
|
|
This is the approximation pattern: `10,000 replay cases` became `600 persisted cases`.
|
|
|
|
### Default fast profile remains far below the allowed 10% scale
|
|
|
|
The plan permits fast CI to run 10% of full gates. Full execution scale is 1,000,000 executions, so a true 10% fast run would be 100,000 executions. The default fast profile still runs 600 executions, which is a smoke-scale run, not a 10% slice.
|
|
|
|
This is the approximation pattern: `10% fast gate` became `small local default profile`.
|
|
|
|
### Perturbation executions are not covered by runtime equivalence
|
|
|
|
The plan says reference/runtime comparison applies to 100% of executions. The base execution is compared against the independent runtime-under-test, but the ten perturbation executions are still evaluated through the reference path for metamorphic statistics. Those perturbation executions are most of the execution workload and are not included in the equivalence gate.
|
|
|
|
This is the approximation pattern: `100% execution comparison` became `base execution comparison`.
|
|
|
|
### Generated world and generated rune-program minimums are not tracked
|
|
|
|
The plan requires 50,000 generated worlds and 250,000 generated rune programs per full CI run. Coverage currently reports generated cases, executions, perturbations, and rejection counts, but not distinct generated worlds or distinct generated rune programs. The required minimums therefore are not directly evidenced.
|
|
|
|
This is the approximation pattern: `explicit generated artifact counts` became `nearby coverage counters`.
|
|
|
|
### Report markdown output is still incomplete
|
|
|
|
The plan requires machine-readable JSON and human-readable markdown for each named report. The implementation emits JSON report files and one combined markdown summary, but not a corresponding markdown report for each required report.
|
|
|
|
This is the approximation pattern: `markdown per required report` became `one summary markdown`.
|
|
|
|
### Collapse analysis still summarizes trace structure rather than reconstructing full traces
|
|
|
|
The collapse implementation is stronger than before because it operates on trace-derived structural features instead of behavior hash proxies. It still reduces each execution to a fixed-width feature row, not the full serialized trace with complete graph topology, edge detail, replay record, faults, and future-state evidence. That leaves a gap between the plan's "trace information" requirement and the current summarized-feature reconstruction.
|
|
|
|
This is the approximation pattern: `full trace information` became `summarized trace feature row`.
|
|
|
|
### Semantic mutation still uses a small fixed mutation corpus
|
|
|
|
The CI pipeline still caps mutation inputs at 64 admitted cases regardless of the configured execution scale. Mutants are now checked against named gates, but survivor detection is still based on a small selected subset rather than the full generated execution corpus.
|
|
|
|
This is the approximation pattern: `mutation checked against the generated execution corpus` became `64 selected cases`.
|
|
|
|
### Mutation pass condition still does not independently enforce the 500-mutant floor
|
|
|
|
The merge-profile provenance checks enforce the mutant floor for merge runs, but `MutationOutcome::passed()` itself still accepts any positive mutant count with zero survivors. The mutation gate remains easy to pass if called outside the merge-profile provenance path with a below-plan count.
|
|
|
|
This is the approximation pattern: `500-mutant acceptance gate` became `positive-count local pass condition`.
|
|
|
|
### Domain read/write checks remain too weak
|
|
|
|
The domain gate still records whether each domain was ever read and ever written at least once. The plan rejects domains that are read-only or write-only across the corpus, but the current evidence can pass a domain that is effectively read-only or write-only except for a token occurrence in one case.
|
|
|
|
This is the approximation pattern: `across-corpus read/write behavior` became `ever observed at least once`.
|
|
|
|
### Domain removal and merging still use behavioral-change proxies
|
|
|
|
Domain removal is still measured through a narrow variation probe and distinct behavior counts. Domain merging is still measured by behavior fingerprint change rate after aliasing domain state. The plan's wording requires corpus behavioral diversity loss for removal and predictive accuracy loss for merging, so these checks remain approximations rather than direct evidence.
|
|
|
|
This is the approximation pattern: `diversity loss and predictive accuracy loss` became `behavior-change proxy`.
|
|
|
|
### Generated case rejection can still return a failed generated case
|
|
|
|
`generate_accepted_case` still has a retry limit and then returns the final generated case even if generated gates are not satisfied. The CI coverage gate can catch admitted failures, but the generator API itself still has a path that violates "Generators must reject flat cases."
|
|
|
|
This is the approximation pattern: `reject flat cases` became `retry then return anyway`.
|
|
|
|
### Semantic contract failures can still be committed on final retry
|
|
|
|
The admission loop still retries contract failures but commits the final attempt after retry exhaustion and records the contract failure afterward. That means the corpus can include a case that does not satisfy its semantic contract, conflicting with the plan's requirement that a case passes only if measured trace behavior satisfies its contract.
|
|
|
|
This is the approximation pattern: `case passes only if contract is satisfied` became `record failure after admission`.
|
|
|
|
### Perturbation response remains absent from individual execution traces
|
|
|
|
`ExecutionTrace` still contains a `perturbation_response` field, but normal runtime resolution leaves it at the default value. Metamorphic evidence is reported at aggregate CI level rather than embedded in the trace object promised by the trace model.
|
|
|
|
This is the approximation pattern: `trace contains perturbation response` became `aggregate report contains perturbation response`.
|
|
|
|
### Neutral perturbation accounting still checks trace neutrality only
|
|
|
|
The unexplained-neutral counter is still based on unchanged trace hash plus missing explanation. The plan's neutral limit is observational neutrality; unchanged delta and unchanged future state are not treated as unexplained neutral when the trace happens to change.
|
|
|
|
This is the approximation pattern: `observational neutrality` became `trace-hash neutrality`.
|