linus-d/magicka-vm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compliance hardening: enforce measured behavior, real gates, retained evidence

Browse Source 
Addresses the attached findings as blocking compliance failures. Each fix
removes a substitution pattern and adds a negative-control test.

Finding 9 (hash changes != enforced metamorphic expectations):
  ci_reports now enforces a sound metamorphic relation per perturbation — a
  perturbation the program CONSUMES must alter the trace — grounded in
  reachability, with a non-vacuity check. Test:
  consumed_perturbation_must_alter_trace.

Finding 10 (trace counts != causal explanation):
  New causal_explanation gate ablates each recorded causal edge's source lane
  and requires the destination delta to change. Measured: true-source 0.77 vs
  scrambled-source 0.03. Threshold 0.50. New required report
  causal_explanation_report. Test:
  causal_edges_are_intervention_confirmed_not_counted.

Finding 6 (static seed corpus != failure retention):
  replay_corpus::retention adds a committed, append-only counterexample corpus
  (retained_failures.tsv) re-verified every run against the independent runtime.
  Negative control: reintroduced_bug_is_caught_by_retention.

Finding 5 (mini mutation evaluator != real acceptance gate):
  Mutants are now killed by ci_reports' OWN acceptance-gate predicates with
  single-sourced thresholds (TRACE_EDGES_MIN, etc.); evaluate_mutants replaces
  semantic_mutation::run_suite on the acceptance path. Test:
  mutants_killed_by_real_acceptance_gates.

Findings 2 & 3 (generated report != independent attestation; merkle root !=
provenance without leaves):
  ci_reports persists every Merkle leaf (evidence/leaves.tsv) + claims. New
  `attestation` crate + `attest` binary recompute the root from the leaves in a
  SEPARATE process that never reads compliance_report.json; wired as a distinct
  merge-gates step. Negative control: tampered_leaf_breaks_attestation.

Finding 8 (structural indicators != measured behavior):
  Domain gate already requires measured influence/mutation/removal; added an
  explicit reject for "appears structurally but no measured influence".

Finding 1 (workflow != merge enforcement): BLOCKED on server-side branch
  protection. Added .github/rulesets/main-required-checks.json + apply command;
  enforcement still requires a repo admin to activate the ruleset.

Finding 7 (protocol socket E2E != rendered browser E2E): BLOCKED on a CI browser
  runner; rendered-browser E2E remains advisory-only.

Finding 4 (trace summary != full trace evidence): PARTIAL. Each retained leaf
  binds the full trace via canonical_hash over all graph edges, and the root is
  independently recomputed from the leaves; per-execution raw-trace round-trip
  reconstruction by the attestor is not yet implemented.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
linus-d
2026-06-21 21:13:48 -07:00
co-authored by Claude Opus 4.8
parent 93c78d9c76
commit 1e50c80627
14 changed files with 1028 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/ci_reports/src/main.rs
+62 -2
View File
@@ -29,6 +29,38 @@ fn write_report(dir: &Path, name: &str, j: &Json) {
f.write_all(j.to_pretty().as_bytes()).expect("write report");
}
/// Write the raw evidence the independent `attest` binary verifies (findings
/// 2, 3): every Merkle leaf, plus the producer's claims. The attestor recomputes
/// the root from these leaves and checks it against the claimed root, in a
/// separate process that never reads the compliance report.
fn write_evidence(dir: &Path, r: &CiResults) {
let ev = dir.join("evidence");
fs::create_dir_all(&ev).expect("create evidence dir");
let mut leaves = String::from("leaf\n");
for h in &r.merkle_leaves {
leaves.push_str(&format!("{:016x}\n", h.0));
}
fs::write(ev.join("leaves.tsv"), leaves).expect("write leaves");
let claims = format!(
"# evidence claims for independent attestation\n\
root\t{:016x}\n\
leaf_count\t{}\n\
total_comparisons\t{}\n\
reference_engine_id\t{:016x}\n\
rut_engine_id\t{:016x}\n\
engines_agree\t{}\n",
r.provenance.execution_merkle_root.0,
r.provenance.merkle_leaf_count,
r.provenance.total_comparisons,
r.provenance.reference_engine_id.0,
r.provenance.rut_engine_id.0,
r.provenance.engines_agree,
);
fs::write(ev.join("claims.tsv"), claims).expect("write claims");
}
fn build_reports(dir: &Path, r: &CiResults) {
// 1. domain_participation_report
write_report(
@@ -111,11 +143,27 @@ fn build_reports(dir: &Path, r: &CiResults) {
("altered_trace".into(), Json::Num(r.metamorphic.altered_trace)),
("altered_delta".into(), Json::Num(r.metamorphic.altered_delta)),
("altered_future".into(), Json::Num(r.metamorphic.altered_future)),
("neutral_unexplained".into(), Json::Num(r.metamorphic.neutral_unexplained)),
("consumed_perturbation_trace_violations".into(), Json::Num(r.metamorphic.expectation_violations)),
("legitimately_neutral".into(), Json::Num(r.metamorphic.explained_neutral)),
("consumed_perturbations".into(), Json::Int(r.metamorphic.consumed as i64)),
("failures".into(), fails(&r.metamorphic.failures)),
]),
);
// 4b. causal_explanation_report (finding 10: intervention-confirmed edges)
write_report(
dir,
"causal_explanation_report",
&Json::Obj(vec![
("pass".into(), pass_field(&r.causal_explanation.failures)),
("edges_tested".into(), Json::Int(r.causal_explanation.edges_tested as i64)),
("edges_confirmed".into(), Json::Int(r.causal_explanation.edges_confirmed as i64)),
("confirmed_fraction".into(), Json::Num(r.causal_explanation.confirmed_fraction)),
("method".into(), Json::s("ablate recorded causal source lane; require destination delta to change")),
("failures".into(), fails(&r.causal_explanation.failures)),
]),
);
// 5. mutation_survivor_report
let survivors: Vec<Json> = r
.mutation
@@ -166,6 +214,9 @@ fn build_reports(dir: &Path, r: &CiResults) {
("total".into(), Json::Int(r.replay.total as i64)),
("deterministic".into(), Json::Int(r.replay.deterministic as i64)),
("drift".into(), Json::Int(r.replay.drift as i64)),
("retained_failures_present".into(), Json::Bool(r.replay.retained_present)),
("retained_failures_total".into(), Json::Int(r.replay.retained_total as i64)),
("retained_failures_regressions".into(), Json::Int(r.replay.retained_regressions as i64)),
("failures".into(), fails(&r.replay.failures)),
]),
);
@@ -226,9 +277,10 @@ fn build_reports(dir: &Path, r: &CiResults) {
/// The reports the spec requires every CI run to produce. A missing or empty
/// artifact is itself an acceptance failure (compliance rule 4).
const REQUIRED_REPORTS: [&str; 9] = [
const REQUIRED_REPORTS: [&str; 10] = [
"domain_participation_report",
"causal_rank_report",
"causal_explanation_report",
"compression_resistance_report",
"metamorphic_response_report",
"mutation_survivor_report",
@@ -338,6 +390,13 @@ fn obligations(r: &CiResults) -> Vec<Obligation> {
actual: 0,
gate_pass: r.metamorphic.failures.is_empty(),
},
Obligation {
requirement: "recorded causal edges are intervention-confirmed (not counted)",
artifact: "causal_explanation_report",
floor: 0,
actual: f(r.causal_explanation.edges_confirmed),
gate_pass: r.causal_explanation.failures.is_empty(),
},
Obligation {
requirement: "every admitted case satisfies its contract",
artifact: "coverage_report",
@@ -555,6 +614,7 @@ fn main() {
let elapsed = start.elapsed();
build_reports(dir, &results);
write_evidence(dir, &results);
write_markdown(dir, &results);
let compliance_ok = build_compliance_report(dir, &results);