Phase 0 · protocol open for review

Proof gates that
don't lie to you.

An agent that can write a proof of a weak specification has produced assurance, not correctness. bendEDA is the harness for a pre-registered study of that gap: Bend 2 laws on one side, a hidden oracle on the other, and a conformance ladder measuring the distance between them.

5
arms, C0-C4
1,800
planned runs
L0-L3
conformance rungs
0
human law authors
ALL PROOFS CHECKbend PROOF.bend
LAWS.bend lockedhash 9f2ac1…
FAR 0.31 → 0.08the question

The problem

Agentic EDA is open-loop

Three failure modes this programme measures rather than assumes away.

Plausible is not correct

Agents emit synthesizable-looking RTL with no deterministic ground truth. Syntax checks pass, synthesis passes, the design is wrong.

Measured by FAR and HFP

A weak spec proves fine

A valid proof of an under-specified law is worthless, and an agent rewarded for closing proofs is rewarded for weakening what is proved.

Measured by RBW, VR, AER

Hidden state, unsafe actions

Long EDA sessions accumulate tool state; an agent can overwrite a golden file or run formal before the RTL exists.

Measured by IAR and UAR

Architecture

The conformance ladder

Bend has no hardware semantics. A proof about a model constrains Verilog only through these links — so each one is a separate, measured gate.

  1. L0

    Laws proved on the model

    bend PROOF.bend must print ALL PROOFS CHECK. This says nothing about RTL, and the site will not pretend otherwise.

    planned
  2. L1

    Trace conformance

    Identical stimulus through model.mjs and the RTL, compared cycle by cycle. Catches divergence on sampled traces.

    planned
  3. L2

    Exhaustive, reduced parameters

    Complete over a 4-bit datapath or a depth-4 FIFO. Complete over a small space is not complete, and is labelled as such.

    planned
  4. L3

    Property transfer

    Laws translated to SVA and proved on the RTL with SymbiYosys. Strongest rung — bounded by translation fidelity, which is itself measured.

    planned
  5. L4

    Bend-to-SystemVerilog emitter

    Removes the translation gap entirely. Out of scope for this study; listed so the gap is a roadmap item, not a footnote.

    research
LAWS.bend→model.bend→ref_model.sv / SVA→RTL

Quickstart

Five minutes, no toolchain

The harness is dependency-free at import time: the CLI and the metric definitions run on a bare Python 3.12 install.

bash
# 1. clone and install
git clone https://github.com/JoeyHammoth/bendEDA
cd bendEDA
python -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"

# 2. check the harness
bendeda --version
bendeda validate --all
python -m unittest discover -s tests -v

Windows

Activate with .venv\Scripts\activate, or open the devcontainer. Bend and the EDA suite run in Linux containers either way.

Stubs exit non-zero

Commands that are not implemented print what they will do and exit 3. Nothing silently passes — that is the whole subject of the study.

Where to read next

README for the shape, tasks/README for a task package, wiki for the running log.

Reference

The bendeda CLI

Every command below exists in the package. “Working” means it does something; “planned” means it exits 3 and tells you which phase implements it. The L1 trace gate is working today — it compiles a Bend model, runs it under Node, and detects an injected one-cycle offset.

17 commands
CommandWhat it doesState
bendeda init [path]Scaffold an experiment or task repositoryworking
bendeda task new <name> --tier BCreate a task package skeleton (9 files, status draft)working
bendeda validate --allValidate shipped schemas and every task packageworking
bendeda conformance compileCompile a Bend model to an ES module at the pinned compiler commitworking
bendeda conformance traceDrive the model cycle by cycle, diff against a DUT, and name the defect (value vs one-cycle offset)working
bendeda law lint <path>Apply rules L001-L015: witnesses, traceability, vacuity, escape hatches, provenanceplanned
bendeda law mutate <model.bend>Mutate the model; every law must be falsified by at least one mutation (LA3)planned
bendeda law translate --to svaTranslate laws through templates and emit a translation-fidelity planplanned
bendeda law provenance <task>Verify the agent authoring chain: run ids, model digests, forbidden readsplanned
bendeda tf run <task>Compute translation fidelity over random and mutated tracesplanned
bendeda canaryRun the checker negative-control suite; one false accept blocks the Bend armsplanned
bendeda gate run <name>Run one gate (lint, sim, formal, conformance) on an artifact directoryplanned
bendeda gate calibrateGate sensitivity and specificity against the seeded-bug corpusplanned
bendeda run <experiment.yaml>Execute an arm × task × model × seed matrix within budgetsplanned
bendeda replay <run_id>Replay a run deterministically from its event logplanned
bendeda oracle seal | verify-commitment | scoreCommit-reveal the sealed oracle, verify the commitment, score a runplanned
bendeda report <experiment>Primary and secondary tables, forest plots, results bundleplanned
bendeda bundle verify <dir>Re-check a proof-carrying design bundle in the pinned imageplanned

Extension surface. Law languages, translators, gates, oracles and hardware boards are Protocols in bendeda/plugins/interfaces.py. A new tool is a plugin, not a fork: the harness never imports a plugin to describe it.

Measurements

Metrics with definitions, not vibes

Each is a pure function of counts, implemented in bendeda/metrics/ and pinned by a hand-computed test. Undefined returns None, never a reassuring 0.0.

FAR

False assurance rate

FP / N

The confirmatory endpoint: runs that looked verified and were wrong, as a share of all runs. Unconditional, so it cannot be gamed by selection.

HFP

Hidden functional pass

Σ O / N

Correctness yield against the hidden oracle. The non-inferiority partner of FAR, so a gate cannot win by rejecting everything.

ER

Escape rate

FP / (TP + FP)

The headline number people quote, reported with n_V because it is conditional on a post-randomisation variable.

TF

Translation fidelity

agreement / traces

How faithfully a Bend law became an RTL property. Admission requires mean ≥ 0.999 and minimum ≥ 0.99.

MKR

Mutation kill ratio

killed / relevant

How much a verification set actually checks, over mutants whose output change is confirmed by equivalence checking.

CPVS

Cost per verified success

Σ tokens / TP

Efficiency of trustworthy output. Undefined with no verified success — which is a finding, not a division error.

Authorship

No human writes the laws

Every artifact in the specification path is written by a separately scoped agent under Freebuff control. The separation of authority that Bend assumes between a spec owner and a prover is preserved exactly; both parties are models.

LAAwrites model and laws; must not read the oracle
RPAwrites reference proofs, different model family
OAAwrites the oracle; must not read the laws
PAwrites the C2 control pack from the spec alone
AAaudits, attacks and signs off LA9-A/B/C
Conductorschedules, locks, merges; authors nothing

The threat this creates, named in public. If the law author and the implementer share a model family, the laws can be self-consistently weak. The protocol answers with cross-family authoring, mutation pressure, an adversarial auditor, an ablation (A7) and a metric (LAW-FAR) — see Appendix J.

Status

Where the programme is

Nothing is frozen until the signed prereg-v1 tag. Before that, every change is a reviewable diff.

  1. Phase 0 · environment and feasibility

    Protocol v0.4, repository foundation, harness skeleton and metric definitions. In progress: pinned images, BendBV, canary suite, sealed-oracle tooling.

  2. Phase 1 · tasks, laws, oracles

    12 task packages, agent-authored law sets, hidden oracles, parity audit.

  3. Phase 2 · harness and tool layer

    MCP servers, policy proxy, runner, replay.

  4. Phase 3 · pilot and freeze

    Calibration, power analysis, signed prereg-v1.

  5. Phases 4-7 · runs, analysis, release

    Longitudinal study, tracks B-D, report, oracle reveal.

Break it, then tell us how

The most useful contribution is a defect in the protocol: a law that admits a bad design, an assumption that excludes legal behaviour, an oracle that misses a plausible bug.