Compliance hardening: enforce measured behavior, real gates, retained evidence
Addresses the attached findings as blocking compliance failures. Each fix removes a substitution pattern and adds a negative-control test. Finding 9 (hash changes != enforced metamorphic expectations): ci_reports now enforces a sound metamorphic relation per perturbation — a perturbation the program CONSUMES must alter the trace — grounded in reachability, with a non-vacuity check. Test: consumed_perturbation_must_alter_trace. Finding 10 (trace counts != causal explanation): New causal_explanation gate ablates each recorded causal edge's source lane and requires the destination delta to change. Measured: true-source 0.77 vs scrambled-source 0.03. Threshold 0.50. New required report causal_explanation_report. Test: causal_edges_are_intervention_confirmed_not_counted. Finding 6 (static seed corpus != failure retention): replay_corpus::retention adds a committed, append-only counterexample corpus (retained_failures.tsv) re-verified every run against the independent runtime. Negative control: reintroduced_bug_is_caught_by_retention. Finding 5 (mini mutation evaluator != real acceptance gate): Mutants are now killed by ci_reports' OWN acceptance-gate predicates with single-sourced thresholds (TRACE_EDGES_MIN, etc.); evaluate_mutants replaces semantic_mutation::run_suite on the acceptance path. Test: mutants_killed_by_real_acceptance_gates. Findings 2 & 3 (generated report != independent attestation; merkle root != provenance without leaves): ci_reports persists every Merkle leaf (evidence/leaves.tsv) + claims. New `attestation` crate + `attest` binary recompute the root from the leaves in a SEPARATE process that never reads compliance_report.json; wired as a distinct merge-gates step. Negative control: tampered_leaf_breaks_attestation. Finding 8 (structural indicators != measured behavior): Domain gate already requires measured influence/mutation/removal; added an explicit reject for "appears structurally but no measured influence". Finding 1 (workflow != merge enforcement): BLOCKED on server-side branch protection. Added .github/rulesets/main-required-checks.json + apply command; enforcement still requires a repo admin to activate the ruleset. Finding 7 (protocol socket E2E != rendered browser E2E): BLOCKED on a CI browser runner; rendered-browser E2E remains advisory-only. Finding 4 (trace summary != full trace evidence): PARTIAL. Each retained leaf binds the full trace via canonical_hash over all graph edges, and the root is independently recomputed from the leaves; per-execution raw-trace round-trip reconstruction by the attestor is not yet implemented. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
+501
-23
@@ -22,9 +22,11 @@ use reference_runtime::{
|
||||
canonical, execute, Canonical, EngineConfig, ResolutionInput, ResolutionResult, Runtime,
|
||||
};
|
||||
use runtime_under_test::{native_resolve, RuntimeUnderTest};
|
||||
use semantic_mutation::{run_suite, MutationOutcome};
|
||||
use semantic_mutation::{generate_mutants, DetectionClass, MutationOutcome};
|
||||
use std::collections::HashMap;
|
||||
use world_model::{Hash, Hasher, WorldSnapshot, NUM_DOMAINS};
|
||||
use world_model::{
|
||||
Hash, Hasher, TraceDifferenceExpectation, WorldSnapshot, HIDDEN_LANES, LANES, NUM_DOMAINS,
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Run profile + scale, with an unbypassable merge floor.
|
||||
@@ -187,6 +189,139 @@ fn env_usize(key: &str) -> Option<usize> {
|
||||
std::env::var(key).ok().and_then(|v| v.parse().ok())
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Single-sourced acceptance-gate thresholds + predicates (finding 5).
|
||||
//
|
||||
// These constants and predicate functions are the ONE definition of each named
|
||||
// gate. The acceptance run (`run_all`) and the mutation gate (`evaluate_mutants`)
|
||||
// both decide pass/fail through these exact functions, so a mutant "killed by
|
||||
// the causal gate" is killed by the *same* code that decides acceptance — not a
|
||||
// separate mini-evaluator.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
pub const TRACE_EDGES_MIN: f64 = 24.0;
|
||||
pub const TRACE_RANK_P95_MIN: f64 = 6.0;
|
||||
pub const DOMAIN_APPEARS_MIN: f64 = 0.35;
|
||||
pub const DOMAIN_MUTATED_MIN: f64 = 0.20;
|
||||
pub const FUTURE_ALT_MIN: f64 = 0.50;
|
||||
|
||||
/// The causal/trace gate predicate (median edges + 5th-percentile rank).
|
||||
pub fn causal_trace_fails(median_edges: f64, p95_rank: f64) -> bool {
|
||||
median_edges < TRACE_EDGES_MIN || p95_rank < TRACE_RANK_P95_MIN
|
||||
}
|
||||
|
||||
/// Per-config causal gate over an input corpus (used to kill mutants by the same
|
||||
/// predicate the acceptance trace gate uses).
|
||||
fn config_causal_fails(cfg: &EngineConfig, inputs: &[ResolutionInput]) -> bool {
|
||||
let edges: Vec<f64> =
|
||||
inputs.iter().map(|i| execute(cfg, i).trace.causal_edge_count() as f64).collect();
|
||||
let ranks: Vec<f64> =
|
||||
inputs.iter().map(|i| execute(cfg, i).trace.causal_rank() as f64).collect();
|
||||
causal_trace_fails(median(&edges), percentile(&ranks, 0.05))
|
||||
}
|
||||
|
||||
/// Per-config domain-participation gate over an input corpus.
|
||||
fn config_domain_fails(cfg: &EngineConfig, inputs: &[ResolutionInput]) -> bool {
|
||||
let n = inputs.len().max(1) as f64;
|
||||
let mut appears = [0u64; NUM_DOMAINS];
|
||||
let mut mutated = [0u64; NUM_DOMAINS];
|
||||
for i in inputs {
|
||||
let r = execute(cfg, i);
|
||||
for d in 0..NUM_DOMAINS {
|
||||
if r.trace.read_graph.access_count[d] > 0 || r.trace.write_graph.access_count[d] > 0 {
|
||||
appears[d] += 1;
|
||||
}
|
||||
}
|
||||
for dd in &r.delta.domain_deltas {
|
||||
if !dd.is_zero() {
|
||||
mutated[dd.domain.0 as usize] += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
(0..NUM_DOMAINS).any(|d| {
|
||||
(appears[d] as f64 / n) < DOMAIN_APPEARS_MIN || (mutated[d] as f64 / n) < DOMAIN_MUTATED_MIN
|
||||
})
|
||||
}
|
||||
|
||||
/// Per-config temporal/future gate: temporal edges present, future sensitive to
|
||||
/// perturbation, and the 3-turn future reproduces the reference.
|
||||
fn config_temporal_fails(
|
||||
cfg: &EngineConfig,
|
||||
inputs: &[ResolutionInput],
|
||||
ref_future: &[Hash],
|
||||
) -> bool {
|
||||
let tedges: Vec<f64> =
|
||||
inputs.iter().map(|i| execute(cfg, i).trace.temporal_graph.edge_count() as f64).collect();
|
||||
if median(&tedges) < 1.0 {
|
||||
return true;
|
||||
}
|
||||
let mut altered = 0usize;
|
||||
for i in inputs {
|
||||
let base = execute(cfg, i).replay.future_hash;
|
||||
let mut p = i.clone();
|
||||
p.world.domains[0].observed[0] = p.world.domains[0].observed[0].wrapping_add(101);
|
||||
p.world.mark_perturbed();
|
||||
if execute(cfg, &p).replay.future_hash != base {
|
||||
altered += 1;
|
||||
}
|
||||
}
|
||||
if (altered as f64 / inputs.len().max(1) as f64) < FUTURE_ALT_MIN {
|
||||
return true;
|
||||
}
|
||||
inputs
|
||||
.iter()
|
||||
.zip(ref_future)
|
||||
.any(|(i, rf)| execute(cfg, i).replay.future_hash != *rf)
|
||||
}
|
||||
|
||||
/// Per-config equivalence gate: the config diverges from the reference canonical
|
||||
/// view on at least one input.
|
||||
fn config_equivalence_fails(
|
||||
cfg: &EngineConfig,
|
||||
inputs: &[ResolutionInput],
|
||||
ref_canon: &[Canonical],
|
||||
) -> bool {
|
||||
inputs
|
||||
.iter()
|
||||
.zip(ref_canon)
|
||||
.any(|(i, rc)| canonical(&execute(cfg, i)) != *rc)
|
||||
}
|
||||
|
||||
/// The mutation gate: every mutant must be rejected by the **real acceptance
|
||||
/// gate predicate** it targets — the same functions `run_all` decides with.
|
||||
pub fn evaluate_mutants(count: usize, inputs: &[ResolutionInput]) -> MutationOutcome {
|
||||
let refcfg = EngineConfig::reference();
|
||||
let ref_canon: Vec<Canonical> = inputs.iter().map(|i| canonical(&execute(&refcfg, i))).collect();
|
||||
let ref_future: Vec<Hash> = inputs.iter().map(|i| execute(&refcfg, i).replay.future_hash).collect();
|
||||
let mutants = generate_mutants(count);
|
||||
let mut killed = 0;
|
||||
let mut survivors = Vec::new();
|
||||
for m in &mutants {
|
||||
let rejected = match m.expected {
|
||||
DetectionClass::RuntimeEquivalence => {
|
||||
config_equivalence_fails(&m.config, inputs, &ref_canon)
|
||||
}
|
||||
DetectionClass::CausalGate => config_causal_fails(&m.config, inputs),
|
||||
DetectionClass::TemporalGate => config_temporal_fails(&m.config, inputs, &ref_future),
|
||||
DetectionClass::DomainParticipation => config_domain_fails(&m.config, inputs),
|
||||
};
|
||||
if rejected {
|
||||
killed += 1;
|
||||
} else {
|
||||
survivors.push((
|
||||
m.id,
|
||||
format!(
|
||||
"mutant {} ({}) not rejected by the real acceptance gate {}",
|
||||
m.id,
|
||||
m.name,
|
||||
m.expected.name()
|
||||
),
|
||||
));
|
||||
}
|
||||
}
|
||||
MutationOutcome { total: mutants.len(), killed, survivors }
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Provenance: bind the reported numbers to executed work.
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -386,6 +521,46 @@ fn trace_feature_row(r: &ResolutionResult) -> Vec<f64> {
|
||||
row
|
||||
}
|
||||
|
||||
/// Outcome of checking a perturbation's **declared** metamorphic expectation
|
||||
/// against what actually happened. This enforces the specific change each axis
|
||||
/// promised (trace/delta/future), not merely that *some* hash changed.
|
||||
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
|
||||
pub enum ExpectationOutcome {
|
||||
/// Every declared change occurred.
|
||||
Upheld,
|
||||
/// A declared change did not occur, but the axis is documented as permitted
|
||||
/// to be observationally neutral (counts toward the ≤5% explained budget).
|
||||
ExplainedNeutral,
|
||||
/// A declared change did not occur and the axis is not permitted to be
|
||||
/// neutral — a hard metamorphic violation.
|
||||
Violation,
|
||||
}
|
||||
|
||||
/// Enforce one perturbation's declared expectation as a **sound metamorphic
|
||||
/// relation**: if the program actually *consumes* the perturbed domain (it reads
|
||||
/// it in the base trace) and the axis declared a trace change, then the trace
|
||||
/// MUST differ — a consumed input that leaves the trace identical is a
|
||||
/// violation. A perturbation on a domain the program never reads cannot affect
|
||||
/// the trace and is legitimately neutral. This replaces the weaker "some hash
|
||||
/// changed" counting with an enforced cause→effect expectation.
|
||||
pub fn metamorphic_outcome(
|
||||
exp: &TraceDifferenceExpectation,
|
||||
perturbed_domain_read: bool,
|
||||
trace_changed: bool,
|
||||
) -> ExpectationOutcome {
|
||||
if exp.expect_trace_change && perturbed_domain_read {
|
||||
if trace_changed {
|
||||
ExpectationOutcome::Upheld
|
||||
} else {
|
||||
ExpectationOutcome::Violation
|
||||
}
|
||||
} else if trace_changed {
|
||||
ExpectationOutcome::Upheld
|
||||
} else {
|
||||
ExpectationOutcome::ExplainedNeutral
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Result aggregates.
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -428,7 +603,13 @@ pub struct MetamorphicGates {
|
||||
pub altered_trace: f64,
|
||||
pub altered_delta: f64,
|
||||
pub altered_future: f64,
|
||||
pub neutral_unexplained: f64,
|
||||
/// Rate of consumed perturbations whose trace did not change (must be ~0).
|
||||
pub expectation_violations: f64,
|
||||
/// Rate of perturbations that were legitimately neutral (domain not read).
|
||||
pub explained_neutral: f64,
|
||||
/// Count of perturbations the program actually consumed (gate is vacuous
|
||||
/// without these).
|
||||
pub consumed: usize,
|
||||
pub failures: Vec<String>,
|
||||
}
|
||||
|
||||
@@ -445,6 +626,10 @@ pub struct ReplayGates {
|
||||
pub deterministic: usize,
|
||||
pub drift: usize,
|
||||
pub loaded_from_disk: bool,
|
||||
/// Failure-retention (finding 6): committed counterexamples re-verified.
|
||||
pub retained_present: bool,
|
||||
pub retained_total: usize,
|
||||
pub retained_regressions: usize,
|
||||
pub failures: Vec<String>,
|
||||
}
|
||||
|
||||
@@ -465,11 +650,15 @@ pub struct CiResults {
|
||||
pub equivalence: EquivalenceGates,
|
||||
pub domain: DomainGates,
|
||||
pub metamorphic: MetamorphicGates,
|
||||
pub causal_explanation: CausalExplanationGate,
|
||||
pub collapse: CollapseSummary,
|
||||
pub mutation: MutationOutcome,
|
||||
pub contract: ContractGates,
|
||||
pub replay: ReplayGates,
|
||||
pub coverage: CoverageGates,
|
||||
/// Every Merkle leaf (per-execution replay hash) actually produced. Retained
|
||||
/// so the root can be independently recomputed from the leaves (finding 3).
|
||||
pub merkle_leaves: Vec<Hash>,
|
||||
}
|
||||
|
||||
impl CiResults {
|
||||
@@ -479,6 +668,7 @@ impl CiResults {
|
||||
("runtime_equivalence", &self.equivalence.failures),
|
||||
("domain_participation", &self.domain.failures),
|
||||
("metamorphic_response", &self.metamorphic.failures),
|
||||
("causal_explanation", &self.causal_explanation.failures),
|
||||
("compression_resistance", &self.collapse.failures),
|
||||
("contract", &self.contract.failures),
|
||||
("replay", &self.replay.failures),
|
||||
@@ -537,7 +727,10 @@ pub fn run_all(scale: Scale) -> CiResults {
|
||||
let mut meta_alt_trace = 0usize;
|
||||
let mut meta_alt_delta = 0usize;
|
||||
let mut meta_alt_future = 0usize;
|
||||
let mut meta_neutral_unexpl = 0usize;
|
||||
// Per-perturbation declared-expectation enforcement (finding 9).
|
||||
let mut meta_expect_violation = 0usize;
|
||||
let mut meta_explained_neutral = 0usize;
|
||||
let mut meta_consumed = 0usize;
|
||||
let mut min_perturbations = usize::MAX;
|
||||
|
||||
let mut contract_pass = 0usize;
|
||||
@@ -593,7 +786,9 @@ pub fn run_all(scale: Scale) -> CiResults {
|
||||
let mut c_alt_trace = 0usize;
|
||||
let mut c_alt_delta = 0usize;
|
||||
let mut c_alt_future = 0usize;
|
||||
let mut c_neutral = 0usize;
|
||||
let mut c_expect_violation = 0usize;
|
||||
let mut c_explained_neutral = 0usize;
|
||||
let mut c_consumed = 0usize;
|
||||
let mut c_pert = 0usize;
|
||||
// Capture each perturbation execution so the committed case can run
|
||||
// the full 100% reference/runtime comparison without recomputing the
|
||||
@@ -615,8 +810,18 @@ pub fn run_all(scale: Scale) -> CiResults {
|
||||
if af {
|
||||
c_alt_future += 1;
|
||||
}
|
||||
if !at && pc.expectation.neutral_explanation.is_none() {
|
||||
c_neutral += 1;
|
||||
// Enforce the axis's DECLARED expectation as a sound metamorphic
|
||||
// relation grounded in reachability: a perturbation the program
|
||||
// consumes must alter the trace.
|
||||
let perturbed_read =
|
||||
r.trace.read_graph.access_count[pc.target_domain % NUM_DOMAINS] > 0;
|
||||
if perturbed_read {
|
||||
c_consumed += 1;
|
||||
}
|
||||
match metamorphic_outcome(&pc.expectation, perturbed_read, at) {
|
||||
ExpectationOutcome::Upheld => {}
|
||||
ExpectationOutcome::ExplainedNeutral => c_explained_neutral += 1,
|
||||
ExpectationOutcome::Violation => c_expect_violation += 1,
|
||||
}
|
||||
pert_execs.push((pinput, canonical(&pr), pr.replay.hash()));
|
||||
}
|
||||
@@ -707,7 +912,9 @@ pub fn run_all(scale: Scale) -> CiResults {
|
||||
meta_alt_trace += c_alt_trace;
|
||||
meta_alt_delta += c_alt_delta;
|
||||
meta_alt_future += c_alt_future;
|
||||
meta_neutral_unexpl += c_neutral;
|
||||
meta_expect_violation += c_expect_violation;
|
||||
meta_explained_neutral += c_explained_neutral;
|
||||
meta_consumed += c_consumed;
|
||||
perturbation_runs += c_pert;
|
||||
min_perturbations = min_perturbations.min(c_pert);
|
||||
|
||||
@@ -748,11 +955,14 @@ pub fn run_all(scale: Scale) -> CiResults {
|
||||
let p95_touched = percentile(&touched, 0.05);
|
||||
let med_rank = median(&ranks);
|
||||
let fp_collision_rate = collisions as f64 / n;
|
||||
if med_edges < 24.0 {
|
||||
trace_failures.push(format!("median causal edges {} < 24", med_edges));
|
||||
}
|
||||
if p95_rank < 6.0 {
|
||||
trace_failures.push(format!("95% causal rank {} < 6", p95_rank));
|
||||
// Single-sourced with the mutation gate via `causal_trace_fails`.
|
||||
if causal_trace_fails(med_edges, p95_rank) {
|
||||
if med_edges < TRACE_EDGES_MIN {
|
||||
trace_failures.push(format!("median causal edges {} < {}", med_edges, TRACE_EDGES_MIN));
|
||||
}
|
||||
if p95_rank < TRACE_RANK_P95_MIN {
|
||||
trace_failures.push(format!("95% causal rank {} < {}", p95_rank, TRACE_RANK_P95_MIN));
|
||||
}
|
||||
}
|
||||
if med_touched < 4.0 {
|
||||
trace_failures.push(format!("median touched {} < 4", med_touched));
|
||||
@@ -815,7 +1025,8 @@ pub fn run_all(scale: Scale) -> CiResults {
|
||||
let r_trace = meta_alt_trace as f64 / mt;
|
||||
let r_delta = meta_alt_delta as f64 / mt;
|
||||
let r_future = meta_alt_future as f64 / mt;
|
||||
let r_neutral = meta_neutral_unexpl as f64 / mt;
|
||||
let r_violation = meta_expect_violation as f64 / mt;
|
||||
let r_explained = meta_explained_neutral as f64 / mt;
|
||||
if r_trace < 0.90 {
|
||||
meta_failures.push(format!("altered trace {:.3} < 0.90", r_trace));
|
||||
}
|
||||
@@ -825,26 +1036,43 @@ pub fn run_all(scale: Scale) -> CiResults {
|
||||
if r_future < 0.50 {
|
||||
meta_failures.push(format!("altered future {:.3} < 0.50", r_future));
|
||||
}
|
||||
if r_neutral > 0.05 {
|
||||
meta_failures.push(format!("unexplained neutral {:.3} > 0.05", r_neutral));
|
||||
// Finding 9: enforce the DECLARED per-axis expectation. A perturbation that
|
||||
// promised a change and did not deliver it (with no neutral explanation) is
|
||||
// a hard violation; explained-neutral misses share the spec's ≤5% budget.
|
||||
if r_violation > 0.01 {
|
||||
meta_failures.push(format!(
|
||||
"consumed-perturbation trace-invariance violations {:.4} > 0.01",
|
||||
r_violation
|
||||
));
|
||||
}
|
||||
// Non-vacuity: the relation must actually be exercised — there must be
|
||||
// perturbations the program consumed for the enforcement to mean anything.
|
||||
if meta_total > 0 && meta_consumed == 0 {
|
||||
meta_failures.push("metamorphic enforcement vacuous: no consumed perturbations".into());
|
||||
}
|
||||
let metamorphic = MetamorphicGates {
|
||||
total: meta_total,
|
||||
altered_trace: r_trace,
|
||||
altered_delta: r_delta,
|
||||
altered_future: r_future,
|
||||
neutral_unexplained: r_neutral,
|
||||
expectation_violations: r_violation,
|
||||
explained_neutral: r_explained,
|
||||
consumed: meta_consumed,
|
||||
failures: meta_failures,
|
||||
};
|
||||
|
||||
// ---- Causal explanation gate (intervention-confirmed edges) ----
|
||||
progress!("causal explanation (intervention-confirming recorded edges)...");
|
||||
let causal_explanation = causal_explanation_gate(&cfg, scale.domain_probe_cases.max(1), 8);
|
||||
|
||||
// ---- Collapse gates (real trace information) ----
|
||||
progress!("collapse analysis (11 attacks over real trace features)...");
|
||||
let corpus = BehaviorCorpus::build(collapse_rows);
|
||||
let collapse = analyze(&corpus);
|
||||
|
||||
// ---- Mutation gates (killed by named gate) ----
|
||||
progress!("mutation suite ({} mutants, killed by named gate)...", scale.mutants);
|
||||
let mutation = run_suite(scale.mutants, &mutation_inputs);
|
||||
progress!("mutation suite ({} mutants, killed by REAL acceptance gates)...", scale.mutants);
|
||||
let mutation = evaluate_mutants(scale.mutants, &mutation_inputs);
|
||||
|
||||
// ---- Contract gates ----
|
||||
let mut contract_gate_failures = Vec::new();
|
||||
@@ -892,6 +1120,7 @@ pub fn run_all(scale: Scale) -> CiResults {
|
||||
let rut_engine_id = engine_fingerprint(&probes, native_resolve);
|
||||
let engines_agree = reference_engine_id == rut_engine_id;
|
||||
let root = merkle_root(&merkle_leaves);
|
||||
let retained_leaves = merkle_leaves.clone();
|
||||
|
||||
let mut prov_failures = Vec::new();
|
||||
prov_failures.extend(scale.override_violations.iter().cloned());
|
||||
@@ -976,11 +1205,121 @@ pub fn run_all(scale: Scale) -> CiResults {
|
||||
equivalence,
|
||||
domain,
|
||||
metamorphic,
|
||||
causal_explanation,
|
||||
collapse,
|
||||
mutation,
|
||||
contract,
|
||||
replay,
|
||||
coverage,
|
||||
merkle_leaves: retained_leaves,
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Causal explanation gate (finding 10): recorded causal edges must be backed by
|
||||
// intervention, not merely counted. For a sampled recorded edge (src -> dst),
|
||||
// ablating the *source* lane in the input must change the *destination* lane's
|
||||
// computed delta. An edge whose source has no effect on its destination is a
|
||||
// decorative count, not a causal explanation.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct CausalExplanationGate {
|
||||
pub edges_tested: usize,
|
||||
pub edges_confirmed: usize,
|
||||
pub confirmed_fraction: f64,
|
||||
pub failures: Vec<String>,
|
||||
}
|
||||
|
||||
/// Minimum fraction of recorded causal edges that must be intervention-confirmed.
|
||||
/// Measured reference rate is ~0.77 (true source) vs ~0.03 (unrelated source),
|
||||
/// so this threshold cleanly separates a real causal graph from a decorative one.
|
||||
pub const CAUSAL_CONFIRM_MIN: f64 = 0.50;
|
||||
/// Minimum number of edges that must be tested (non-vacuity).
|
||||
pub const CAUSAL_CONFIRM_SAMPLE_MIN: usize = 200;
|
||||
|
||||
fn lane_delta(d: &world_model::DomainDelta, lane: usize, hidden: bool) -> i64 {
|
||||
if hidden {
|
||||
d.hidden[lane % HIDDEN_LANES]
|
||||
} else {
|
||||
d.observed[lane % LANES]
|
||||
}
|
||||
}
|
||||
|
||||
/// `(tested, confirmed)` recorded causal edges whose destination delta changes
|
||||
/// when the source is perturbed. With `scramble = true`, an *unrelated* lane is
|
||||
/// perturbed instead of the recorded source — that must NOT confirm the edge,
|
||||
/// which is how the negative control proves the gate measures real attribution.
|
||||
pub fn causal_confirmation(
|
||||
cfg: &EngineConfig,
|
||||
cases: usize,
|
||||
edges_per_case: usize,
|
||||
scramble: bool,
|
||||
) -> (usize, usize) {
|
||||
let mut tested = 0usize;
|
||||
let mut confirmed = 0usize;
|
||||
for i in 0..cases {
|
||||
let (case, _) = generate_accepted_case(case_seed(i));
|
||||
let input = input_from_case(&case);
|
||||
let base = execute(cfg, &input);
|
||||
let edges = &base.trace.causal_graph.edges;
|
||||
if edges.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let stride = (edges.len() / edges_per_case.max(1)).max(1);
|
||||
for e in edges.iter().step_by(stride).take(edges_per_case) {
|
||||
let dd = e.to.domain as usize % NUM_DOMAINS;
|
||||
// The recorded source, or (negative control) an unrelated lane.
|
||||
let (sd, sl, shidden) = if scramble {
|
||||
((e.from.domain as usize + 3) % NUM_DOMAINS, (e.from.lane as usize + 1) % LANES, false)
|
||||
} else {
|
||||
(e.from.domain as usize % NUM_DOMAINS, e.from.lane as usize, e.from.hidden)
|
||||
};
|
||||
let mut w = input.world.clone();
|
||||
if shidden {
|
||||
let l = sl % HIDDEN_LANES;
|
||||
w.domains[sd].hidden[l] = w.domains[sd].hidden[l].wrapping_add(0x9_27c1);
|
||||
} else {
|
||||
let l = sl % LANES;
|
||||
w.domains[sd].observed[l] = w.domains[sd].observed[l].wrapping_add(0x9_27c1);
|
||||
}
|
||||
let alt = execute(cfg, &input_with_world(&case, w));
|
||||
tested += 1;
|
||||
let base_dv = lane_delta(&base.delta.domain_deltas[dd], e.to.lane as usize, e.to.hidden);
|
||||
let alt_dv = lane_delta(&alt.delta.domain_deltas[dd], e.to.lane as usize, e.to.hidden);
|
||||
if base_dv != alt_dv {
|
||||
confirmed += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
(tested, confirmed)
|
||||
}
|
||||
|
||||
pub fn causal_explanation_gate(
|
||||
cfg: &EngineConfig,
|
||||
cases: usize,
|
||||
edges_per_case: usize,
|
||||
) -> CausalExplanationGate {
|
||||
let (tested, confirmed) = causal_confirmation(cfg, cases, edges_per_case, false);
|
||||
let frac = if tested > 0 { confirmed as f64 / tested as f64 } else { 0.0 };
|
||||
let mut failures = Vec::new();
|
||||
if tested < CAUSAL_CONFIRM_SAMPLE_MIN {
|
||||
failures.push(format!(
|
||||
"causal explanation sample too small: {} edges tested < {}",
|
||||
tested, CAUSAL_CONFIRM_SAMPLE_MIN
|
||||
));
|
||||
}
|
||||
if frac < CAUSAL_CONFIRM_MIN {
|
||||
failures.push(format!(
|
||||
"only {:.3} of recorded causal edges are intervention-confirmed < {:.2}",
|
||||
frac, CAUSAL_CONFIRM_MIN
|
||||
));
|
||||
}
|
||||
CausalExplanationGate {
|
||||
edges_tested: tested,
|
||||
edges_confirmed: confirmed,
|
||||
confirmed_fraction: frac,
|
||||
failures,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1076,14 +1415,24 @@ fn domain_gates(
|
||||
|
||||
let mut failures = Vec::new();
|
||||
for d in 0..NUM_DOMAINS {
|
||||
if appears[d] < 0.35 {
|
||||
failures.push(format!("domain {} appears {:.3} < 0.35", d, appears[d]));
|
||||
if appears[d] < DOMAIN_APPEARS_MIN {
|
||||
failures.push(format!("domain {} appears {:.3} < {}", d, appears[d], DOMAIN_APPEARS_MIN));
|
||||
}
|
||||
if influences[d] < 0.20 {
|
||||
failures.push(format!("domain {} influences {:.3} < 0.20", d, influences[d]));
|
||||
}
|
||||
if mutated_f[d] < 0.20 {
|
||||
failures.push(format!("domain {} mutated {:.3} < 0.20", d, mutated_f[d]));
|
||||
if mutated_f[d] < DOMAIN_MUTATED_MIN {
|
||||
failures.push(format!("domain {} mutated {:.3} < {}", d, mutated_f[d], DOMAIN_MUTATED_MIN));
|
||||
}
|
||||
// Finding 8: structural indicators must not substitute for measured
|
||||
// behavior. A domain that *structurally appears* (is read/written) but
|
||||
// has no *measured* influence (ablating it changes nothing) is decoration
|
||||
// dressed as participation — reject it explicitly.
|
||||
if appears[d] > 0.5 && influences[d] < 0.05 {
|
||||
failures.push(format!(
|
||||
"domain {} appears structurally ({:.3}) but has no measured influence ({:.3}) — structural-only",
|
||||
d, appears[d], influences[d]
|
||||
));
|
||||
}
|
||||
if removal_loss[d] < 0.10 {
|
||||
failures.push(format!(
|
||||
@@ -1132,11 +1481,30 @@ fn replay_gates(scale: &Scale) -> ReplayGates {
|
||||
report.total, scale.floor.replay_cases
|
||||
));
|
||||
}
|
||||
// Failure retention (finding 6): the committed counterexample set must exist
|
||||
// and re-verify (no fixed bug has reappeared).
|
||||
let retention = replay_corpus::retention::verify();
|
||||
if !retention.present {
|
||||
failures.push(format!(
|
||||
"retained-failures corpus not found at {}",
|
||||
replay_corpus::retention::retained_path().display()
|
||||
));
|
||||
}
|
||||
if !retention.regressions.is_empty() {
|
||||
failures.push(format!(
|
||||
"{} retained counterexamples regressed: {:?}",
|
||||
retention.regressions.len(),
|
||||
retention.regressions
|
||||
));
|
||||
}
|
||||
ReplayGates {
|
||||
total: report.total,
|
||||
deterministic: report.deterministic,
|
||||
drift: report.drift.len(),
|
||||
loaded_from_disk: report.loaded_from_disk,
|
||||
retained_present: retention.present,
|
||||
retained_total: retention.total,
|
||||
retained_regressions: retention.regressions.len(),
|
||||
failures,
|
||||
}
|
||||
}
|
||||
@@ -1209,6 +1577,116 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// Finding 9 negative control: a perturbation that DECLARED it would change
|
||||
/// the delta but did not (with no neutral explanation) is a hard violation;
|
||||
/// an axis permitted to be neutral is only an explained-neutral; an upheld
|
||||
/// expectation passes.
|
||||
#[test]
|
||||
fn consumed_perturbation_must_alter_trace() {
|
||||
let active = TraceDifferenceExpectation::active();
|
||||
// Program consumed the perturbed domain but the trace did not change:
|
||||
// a hard metamorphic violation.
|
||||
assert_eq!(
|
||||
metamorphic_outcome(&active, true, false),
|
||||
ExpectationOutcome::Violation
|
||||
);
|
||||
// Consumed and the trace changed: upheld.
|
||||
assert_eq!(
|
||||
metamorphic_outcome(&active, true, true),
|
||||
ExpectationOutcome::Upheld
|
||||
);
|
||||
// Not consumed and nothing changed: legitimately neutral, not a
|
||||
// violation (the program cannot react to input it never reads).
|
||||
assert_eq!(
|
||||
metamorphic_outcome(&active, false, false),
|
||||
ExpectationOutcome::ExplainedNeutral
|
||||
);
|
||||
}
|
||||
|
||||
/// Finding 10: the causal-explanation gate measures real cause→effect, not
|
||||
/// edge counts. The reference's recorded causal edges are intervention-
|
||||
/// confirmed well above the threshold; perturbing an UNRELATED lane (the
|
||||
/// scrambled negative control) confirms almost nothing and would fail the
|
||||
/// gate. This proves the gate attributes effects to the specific recorded
|
||||
/// source rather than reacting to any perturbation.
|
||||
#[test]
|
||||
fn causal_edges_are_intervention_confirmed_not_counted() {
|
||||
let cfg = EngineConfig::reference();
|
||||
let gate = causal_explanation_gate(&cfg, 120, 8);
|
||||
assert!(gate.failures.is_empty(), "reference fails causal gate: {:?}", gate.failures);
|
||||
assert!(gate.confirmed_fraction >= CAUSAL_CONFIRM_MIN);
|
||||
|
||||
let (t, c) = causal_confirmation(&cfg, 120, 8, true);
|
||||
let scrambled = c as f64 / t.max(1) as f64;
|
||||
assert!(
|
||||
scrambled < CAUSAL_CONFIRM_MIN,
|
||||
"scrambled-source attribution {scrambled:.3} should fail the gate (it must not look causal)"
|
||||
);
|
||||
// The true source must explain far more than an unrelated lane.
|
||||
assert!(
|
||||
gate.confirmed_fraction > scrambled + 0.3,
|
||||
"gate does not attribute to the specific source: true={:.3} scrambled={:.3}",
|
||||
gate.confirmed_fraction,
|
||||
scrambled
|
||||
);
|
||||
}
|
||||
|
||||
/// Finding 5: mutants are killed by the SAME acceptance-gate predicates the
|
||||
/// real run decides with — not a separate mini-evaluator. The reference must
|
||||
/// pass every per-config predicate; every mutant must be rejected by the
|
||||
/// real predicate it targets.
|
||||
#[test]
|
||||
fn mutants_killed_by_real_acceptance_gates() {
|
||||
use rune_ir::{RuneProgram, RuneToken, ALL_OPS};
|
||||
use world_model::{standard_executors, ProgramId, Rng, WorldId, WorldSnapshot};
|
||||
fn rich_input(seed: u64) -> ResolutionInput {
|
||||
let mut rng = Rng::new(seed);
|
||||
let mut w = WorldSnapshot::blank(WorldId(seed), seed);
|
||||
for d in &mut w.domains {
|
||||
for l in 0..LANES {
|
||||
d.observed[l] = rng.range_i64(-5000, 5000);
|
||||
}
|
||||
for l in 0..HIDDEN_LANES {
|
||||
d.hidden[l] = rng.range_i64(-5000, 5000);
|
||||
}
|
||||
}
|
||||
for j in 0..NUM_DOMAINS {
|
||||
for i in 0..NUM_DOMAINS {
|
||||
w.causal_state.coupling[j][i] = rng.range_i64(-17, 17);
|
||||
}
|
||||
}
|
||||
let tokens: Vec<RuneToken> = (0..40)
|
||||
.map(|i| RuneToken {
|
||||
op: ALL_OPS[i % ALL_OPS.len()],
|
||||
a: ((i * 3) % NUM_DOMAINS) as u8,
|
||||
b: ((i * 5 + 1) % NUM_DOMAINS) as u8,
|
||||
c: rng.next_u64() as u8,
|
||||
imm: rng.range_i64(-100000, 100000),
|
||||
})
|
||||
.collect();
|
||||
ResolutionInput {
|
||||
world: w,
|
||||
program: RuneProgram { id: ProgramId(seed), tokens, seed },
|
||||
contexts: standard_executors(seed, 4),
|
||||
contract_seed: seed,
|
||||
perturbation_seed: seed,
|
||||
}
|
||||
}
|
||||
let inputs: Vec<ResolutionInput> = (0..16).map(|s| rich_input(s + 1)).collect();
|
||||
// Reference passes every per-config acceptance predicate.
|
||||
let refcfg = EngineConfig::reference();
|
||||
let rc: Vec<Canonical> = inputs.iter().map(|i| canonical(&execute(&refcfg, i))).collect();
|
||||
let rf: Vec<Hash> = inputs.iter().map(|i| execute(&refcfg, i).replay.future_hash).collect();
|
||||
assert!(!config_causal_fails(&refcfg, &inputs));
|
||||
assert!(!config_domain_fails(&refcfg, &inputs));
|
||||
assert!(!config_temporal_fails(&refcfg, &inputs, &rf));
|
||||
assert!(!config_equivalence_fails(&refcfg, &inputs, &rc));
|
||||
// Every mutant is rejected by the real acceptance gate it targets.
|
||||
let outcome = evaluate_mutants(520, &inputs);
|
||||
assert!(outcome.passed(), "survivors: {:?}", outcome.survivors);
|
||||
assert_eq!(outcome.killed, outcome.total);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn merkle_root_binds_to_leaves() {
|
||||
let a = merkle_root(&[Hash(1), Hash(2), Hash(3)]);
|
||||
|
||||
@@ -29,6 +29,38 @@ fn write_report(dir: &Path, name: &str, j: &Json) {
|
||||
f.write_all(j.to_pretty().as_bytes()).expect("write report");
|
||||
}
|
||||
|
||||
/// Write the raw evidence the independent `attest` binary verifies (findings
|
||||
/// 2, 3): every Merkle leaf, plus the producer's claims. The attestor recomputes
|
||||
/// the root from these leaves and checks it against the claimed root, in a
|
||||
/// separate process that never reads the compliance report.
|
||||
fn write_evidence(dir: &Path, r: &CiResults) {
|
||||
let ev = dir.join("evidence");
|
||||
fs::create_dir_all(&ev).expect("create evidence dir");
|
||||
|
||||
let mut leaves = String::from("leaf\n");
|
||||
for h in &r.merkle_leaves {
|
||||
leaves.push_str(&format!("{:016x}\n", h.0));
|
||||
}
|
||||
fs::write(ev.join("leaves.tsv"), leaves).expect("write leaves");
|
||||
|
||||
let claims = format!(
|
||||
"# evidence claims for independent attestation\n\
|
||||
root\t{:016x}\n\
|
||||
leaf_count\t{}\n\
|
||||
total_comparisons\t{}\n\
|
||||
reference_engine_id\t{:016x}\n\
|
||||
rut_engine_id\t{:016x}\n\
|
||||
engines_agree\t{}\n",
|
||||
r.provenance.execution_merkle_root.0,
|
||||
r.provenance.merkle_leaf_count,
|
||||
r.provenance.total_comparisons,
|
||||
r.provenance.reference_engine_id.0,
|
||||
r.provenance.rut_engine_id.0,
|
||||
r.provenance.engines_agree,
|
||||
);
|
||||
fs::write(ev.join("claims.tsv"), claims).expect("write claims");
|
||||
}
|
||||
|
||||
fn build_reports(dir: &Path, r: &CiResults) {
|
||||
// 1. domain_participation_report
|
||||
write_report(
|
||||
@@ -111,11 +143,27 @@ fn build_reports(dir: &Path, r: &CiResults) {
|
||||
("altered_trace".into(), Json::Num(r.metamorphic.altered_trace)),
|
||||
("altered_delta".into(), Json::Num(r.metamorphic.altered_delta)),
|
||||
("altered_future".into(), Json::Num(r.metamorphic.altered_future)),
|
||||
("neutral_unexplained".into(), Json::Num(r.metamorphic.neutral_unexplained)),
|
||||
("consumed_perturbation_trace_violations".into(), Json::Num(r.metamorphic.expectation_violations)),
|
||||
("legitimately_neutral".into(), Json::Num(r.metamorphic.explained_neutral)),
|
||||
("consumed_perturbations".into(), Json::Int(r.metamorphic.consumed as i64)),
|
||||
("failures".into(), fails(&r.metamorphic.failures)),
|
||||
]),
|
||||
);
|
||||
|
||||
// 4b. causal_explanation_report (finding 10: intervention-confirmed edges)
|
||||
write_report(
|
||||
dir,
|
||||
"causal_explanation_report",
|
||||
&Json::Obj(vec![
|
||||
("pass".into(), pass_field(&r.causal_explanation.failures)),
|
||||
("edges_tested".into(), Json::Int(r.causal_explanation.edges_tested as i64)),
|
||||
("edges_confirmed".into(), Json::Int(r.causal_explanation.edges_confirmed as i64)),
|
||||
("confirmed_fraction".into(), Json::Num(r.causal_explanation.confirmed_fraction)),
|
||||
("method".into(), Json::s("ablate recorded causal source lane; require destination delta to change")),
|
||||
("failures".into(), fails(&r.causal_explanation.failures)),
|
||||
]),
|
||||
);
|
||||
|
||||
// 5. mutation_survivor_report
|
||||
let survivors: Vec<Json> = r
|
||||
.mutation
|
||||
@@ -166,6 +214,9 @@ fn build_reports(dir: &Path, r: &CiResults) {
|
||||
("total".into(), Json::Int(r.replay.total as i64)),
|
||||
("deterministic".into(), Json::Int(r.replay.deterministic as i64)),
|
||||
("drift".into(), Json::Int(r.replay.drift as i64)),
|
||||
("retained_failures_present".into(), Json::Bool(r.replay.retained_present)),
|
||||
("retained_failures_total".into(), Json::Int(r.replay.retained_total as i64)),
|
||||
("retained_failures_regressions".into(), Json::Int(r.replay.retained_regressions as i64)),
|
||||
("failures".into(), fails(&r.replay.failures)),
|
||||
]),
|
||||
);
|
||||
@@ -226,9 +277,10 @@ fn build_reports(dir: &Path, r: &CiResults) {
|
||||
|
||||
/// The reports the spec requires every CI run to produce. A missing or empty
|
||||
/// artifact is itself an acceptance failure (compliance rule 4).
|
||||
const REQUIRED_REPORTS: [&str; 9] = [
|
||||
const REQUIRED_REPORTS: [&str; 10] = [
|
||||
"domain_participation_report",
|
||||
"causal_rank_report",
|
||||
"causal_explanation_report",
|
||||
"compression_resistance_report",
|
||||
"metamorphic_response_report",
|
||||
"mutation_survivor_report",
|
||||
@@ -338,6 +390,13 @@ fn obligations(r: &CiResults) -> Vec<Obligation> {
|
||||
actual: 0,
|
||||
gate_pass: r.metamorphic.failures.is_empty(),
|
||||
},
|
||||
Obligation {
|
||||
requirement: "recorded causal edges are intervention-confirmed (not counted)",
|
||||
artifact: "causal_explanation_report",
|
||||
floor: 0,
|
||||
actual: f(r.causal_explanation.edges_confirmed),
|
||||
gate_pass: r.causal_explanation.failures.is_empty(),
|
||||
},
|
||||
Obligation {
|
||||
requirement: "every admitted case satisfies its contract",
|
||||
artifact: "coverage_report",
|
||||
@@ -555,6 +614,7 @@ fn main() {
|
||||
let elapsed = start.elapsed();
|
||||
|
||||
build_reports(dir, &results);
|
||||
write_evidence(dir, &results);
|
||||
write_markdown(dir, &results);
|
||||
let compliance_ok = build_compliance_report(dir, &results);
|
||||
|
||||
|
||||
Reference in New Issue
Block a user