Vision

Vaibify is a harness for making computational science reproducible in a world where AI agents are first-class participants in the research process. This document describes what that means, the graded framework we use to reason about reproducibility, and where vaibify sits on it.

For the motivation that preceded vaibify — why secure containment and agent-assisted computing matter — see philosophy.md. For the day-to-day mechanics of the reproducibility stack, see reproducibility.md. This document is intentionally higher-level and forward-looking, so that it can be cited independently of any single vaibify release.

The problem

Two concurrent shifts make the existing reproducibility story insufficient.

Agentic contributors. AI systems now produce plausible scientific output at speeds that overwhelm peer review and without the informal trust signals (reputation, institution, career-staking) that human science relies on. A paper generated by an agent has no career at stake; its reliability has to come from the work itself, not the author. See EviBound (arXiv 2511.05524) for the architectural framing and REPRO-Bench (arXiv 2507.18901) for empirical evidence that agents cannot reliably self-assess reproducibility.

Cryptographic possibility. The building blocks for machine-verifiable scientific claims now exist in production — content-addressed storage (git), immutable archival (Zenodo DOIs, Software Heritage), supply-chain attestation (Sigstore, SLSA), deterministic container builds, and in the longer term zero-knowledge proofs of computation. None of this infrastructure was realistic for working scientists a decade ago.

The opening is therefore to treat a reproducible scientific artifact not as a PDF plus some code, but as a content-addressed chain of evidence whose integrity any third party — human or agent — can verify without rerunning the computation.

The PROOF ladder

Reproducibility is not a binary. We distinguish six levels, each strictly harder than the last. The ladder is named for the five pillars a result must rest on to be trusted by someone who was not there: Provenance, Reproducibility, Openness, Oversight, Falsifiability. The structure follows the pattern used in other trust-across-boundaries domains: SLSA in software supply-chain security, evidence-based medicine hierarchies, NIST metrological traceability, art-world provenance. At each level we state what it proves and, equally important, what it does not.

L1 — Self-Consistent

All project tests pass. Every file declared as canonical matches the workspace’s last local commit. Test markers reproduce on a fresh clone: the recorded content hashes match what the author verified.

  • Proves: the author has not silently modified anything between verification and hand-off.

  • Does not prove: that the author’s state is consistent with anything outside the author’s own machine.

L2 — Published

Every canonical file’s content hash matches what is published at a specific, immutable remote authority: GitHub’s blob at a committed commit SHA; Overleaf’s referenced revision; the Zenodo DOI’s archive for bundled data. The commit SHA is reachable on the public branch.

  • Proves: a third party can verify the published record matches the author’s container using only hash fetches from public authorities. No trust in the author’s environment required.

  • Does not prove: that the computation producing those files can be re-executed to yield the same bits.

A project that intends to reach L5 one day publishes its equivalence tolerances here, in the paper, because L5 requires them to predate the regeneration they will judge. L2 itself does not require them — the gate is the hash agreement above, and a project with no ambition beyond publication owes no tolerance table. This is the same shape as L4’s transcript: recorded early because a later rung needs it to already exist, not because this rung asks for it.

L3 — Reproducible

The Docker image is reproducibly buildable: docker build . at the committed Dockerfile produces a byte-identical image hash on any capable host. The project is deterministic — random seeds declared, BLAS threading pinned, no nondeterministic libraries. Running vaibify reproduce <url> on a fresh clone regenerates output files whose hashes match the committed baseline.

  • Proves: a third party can regenerate the author’s outputs from source + inputs and get identical bits.

  • Does not prove: that the inputs themselves are authentic.

The jump L2 → L3 is genuinely hard. Deterministic numerics on modern hardware requires care (OMP_NUM_THREADS=1 or MKL_CBWR=COMPATIBLE, avoidance of nondeterministic CUDA operations); reproducible Docker builds require pinned apt versions and SOURCE_DATE_EPOCH. Much of this tooling exists in the reproducible-builds community; vaibify’s role is to package it for working scientists.

L4 — Traceable

The whole sequence of events, from the first commit to the curation of a long-term archive, is publicly recorded: the raw data, the analysis scripts, the plotting scripts behind every published figure, and a public changelog for each of them. The changelog is two things together — the git commit history and the transcript of the AI prompts that produced the work — because with agentic contributors neither half alone says who decided what. Every external input is recorded with (source URL, fetch timestamp, content hash) in a committed manifest; re-fetching the URL produces a matching hash, and archival services (Zenodo, Software Heritage, Wayback Machine) hold snapshots in case the source disappears. Traceability runs back through calibration to the raw observations themselves.

The rung is named for the unbroken chain, not the copies that back it up: what L4 adds is metrological traceability — every input and every decision is followed back to a documented origin, the same relation that anchors a measurement to a reference standard or a painting to its provenance. L3 proves you can regenerate the outputs; L4 proves you can vouch for what went in and for how it got there. (It was called “Archived” until 2026-08, which named a mechanism rather than the claim, and named one that L2 already uses: a Zenodo DOI is a publication authority there.)

L4 is the one rung that cannot be reached retroactively. The prompt transcript has to exist before L1, so a project that did not plan for L4 at conception cannot climb to it later. That is why vaibify records AI provenance on its own axis from the beginning — see The Replay axis below — even though the ladder rung itself is out of scope.

  • Proves: the full causal chain from raw observation to published plot is tamper-evident, and the development history that produced it is public.

  • Does not prove: that the result is independent of the particular agents that produced it.

L5 — Regenerated

The work is retraced by a different AI model and the result comes out the same. This controls for the biases and assumptions baked into whichever model wrote the original code.

The regeneration starts from the publication, not the transcript. The L2 paper — its hypothesis, data, methods, and acceptance criteria — is the specification, and it is the right one on three counts: it already exists, it is immutable and timestamped by a third party, and reproducing from the published methods is what replication has always meant. The L4 transcript stays public for audit; it is deliberately not the input, because replaying it reproduces the first model’s path rather than independently deriving anything.

The equivalence tolerance is declared at L2, in the publication. A table states, per compared quantity, how close is close enough and why — “0.1 K, a tenth of the model’s grid resolution” rather than a bare “0.1 K” — and it must cover every quantity the paper’s claims rest on. Tolerances published for three quantities out of thirty yield a verdict over a curated subset: honestly computed, and close to meaningless. Because the table is published before any regeneration exists, the threshold is pre-registered. The judgement remains human and qualitative; what it can no longer be is made with knowledge of the answer.

L5 then returns a verdict: did every compared quantity fall inside the published tolerance? That is mechanical, and anyone holding both artifacts can compute it. What stays with the reader is whether the tolerance was appropriate — and because it is public, a skeptic who finds it too loose applies their own and reaches their own conclusion from the same material. The ladder lays out the evidence; it does not adjudicate the threshold.

Byte-identity across the two runs is neither expected nor required. Different models write different code with different orders of operation, so the artifacts will usually differ in their last significant figures. What is required is that each artifact independently satisfies L3, so both are internally bit-reproducible and a third party can verify either one exactly.

Two limits are worth stating rather than discovering. First, “a different model” means a different model identifier, which is a floor rather than a guarantee. Vendors do not disclose whether two of their models share a pretraining corpus, so a stricter criterion would be unverifiable by anyone outside the vendor — and a rung that depends on undisclosed information is uncheckable by construction. Same-vendor siblings very likely do share one, which weakens the independence claim considerably; a cross-vendor regeneration, and better still an open-weight one, is substantially stronger. The models used are declared either way, so a reader can judge a particular pairing.

Second, the specification covers the science, not the software. A regeneration produces a different codebase with different steps, so only output quantities are comparable — which is why the tolerance is declared per quantity rather than per file. It also means L5 costs a second complete L3 envelope: another manifest, lockfile, pinned image digest, and immutable deposit.

A regeneration that fails because the published methods were not sufficient to work from is not wasted effort. It is a public, mechanical demonstration that the methods section was inadequate, which is a more useful finding than most of what a passing L5 establishes.

  • Proves: the result does not depend on the particular model that produced it, to within a threshold fixed before the answer was known.

  • Does not prove: that the result is independent of the researcher — the same person directed both runs and cannot un-know the first outcome. Nor that anyone besides the author has verified either run.

L6 — Attested

All of the above, plus independent and publicly verifiable attestations by trusted third parties — collaborators, CI services, archival bots — who have retraced the work themselves and confirmed the L3 reproducibility of both artifacts along with the equivalence verdict between them. Note what is not claimed: attestors do not report byte-identical results across the original and the regeneration, which L5 has already established is not the expectation. They verify each artifact exactly, and the equivalence judgement against the published tolerance.

The attestations must be verifiable in public: tests run on a public GitHub repository, or cryptographic links to witness identities (ORCID keys bound to the DOIs of remote archives). Zenodo supports this directly, and standalone tooling exists in in-toto and the Sigstore / Rekor transparency logs.

L6 is also where independence from the researcher arrives. Its attestors never saw the original process, so they control for the human as well as the model — which is the thing L5 structurally cannot do, since one person directs both of its runs.

L6 is a high bar that costs the community real effort, so in the near term it will be reserved for results that warrant it — the claim of life on another world, not a routine parameter study.

  • Proves: reproducibility is no longer the author’s claim alone; it is community-verified, and the community verification is itself publicly auditable.

  • Does not prove: that the result describes the universe correctly. Corrupt observations, conceptual errors, and a bug that every agent makes identically all survive L6 intact. This is the ceiling of what hashes and signatures alone can guarantee.

What this framework is and is not

The framework is a vocabulary for stating, and checking, the reproducibility rigor of a computational result. It is strictly independent of vaibify. Any tool that implements container packaging, external hash fetches, reproducible builds, and attestation logs could claim levels on it. Vaibify is an implementation, not the implementation.

Two properties we deliberately keep off the ladder:

  • Test meaningfulness. Whether a project’s tests are actually strong enough to catch subtle errors (mutation-testing coverage, adversarial robustness) is a property a project should have at any level. It is a property of the tests, not of reproducibility.

  • Physics-informed validation. Whether the numerical result respects conservation laws, symmetries, or asymptotic limits is a correctness claim about the science, not a reproducibility claim about the bits. Results can be perfectly reproducible and scientifically wrong. L5’s verdict is not an exception to this: it checks a threshold the author fixed in advance and published, and whether that threshold was the right one is itself off the ladder, left to the reader who can see it.

Both deserve attention; neither is a rung on this ladder.

Where vaibify sits

Vaibify implements L1, L2, and L3, and stops there by design. All three have working enforcement in vaibify/reproducibility/levelGates.py: fbAtLeastLevel1, _fbComputeLevel2’s seven conjuncts, and fbAtLeastLevel3 (L2 plus seven readiness verifiers plus a current attestation), each with a blocker list that tells the researcher exactly what is missing.

L4, L5, and L6 are described here for completeness and are not implemented: they depend on institutional archiving, cross-agent regeneration, and third-party attestation that sit outside a local-first tool. See reproducibility.md, which is the normative statement of the L3 ceiling.

One further limit is worth stating here because it surprises people: Level 3 requires a containerized project. A host-mode project — one whose pipeline runs directly on the researcher’s own machine — is refused L3 with the host-mode criterion, because the level is defined by a pinned image digest and an in-container rerun, and a host project has neither. Host mode reaches L2 and stops.

Two deliberate design choices shape the implementation.

Local-first. The workspace is a directory on the scientist’s own machine, mounted into a container; artifacts stay on the scientist’s disk and under the scientist’s control. Contrast with hosted platforms (Whole Tale, Renku, Code Ocean) that require uploading work to run it. Local-first is more architecturally demanding — notably, workspace storage varies across host operating systems — but it respects how working scientists already organize their work and avoids creating yet another platform lock-in.

AI agents as equal participants in the authorship graph. Infrastructure that assumes “the author” is a human and “tools” are what agents use cannot cleanly accommodate a future where significant portions of a project are produced by agents. Vaibify’s pre-push manifest check, content-hash-bound test markers, and per-remote freshness badges exist so that agent output is subject to the same verification gates as human output, with no special privilege in either direction.

The Replay axis

The ladder measures the state of the artifact. It says nothing about the provenance of the process that produced it, which is a separate question once agents are doing the writing — and one whose evidence must be collected from the very beginning, since a prompt transcript cannot be reconstructed after the fact.

Vaibify therefore tracks AI provenance on its own axis, orthogonal to the ladder, in vaibify/reproducibility/replayGate.py. Its states, in ascending order:

  • untracked — nothing declared.

  • declared — every model used in the project is declared, with vendor, model id, and the interval it was used over. This is the one state the ladder also depends on: L2 requires it.

  • recorded — the Prompt Record is enabled and its first capture reviewed. The in-container agent’s session transcripts are copied into the repository under .vaibify/promptRecord/, sanitized at capture (nothing lands in a public repository unscanned) and hash-chained, so removing or editing a record breaks the chain. A project is “Replayable” at this state or better.

  • supervised — the attribution watchdog is enabled, recording file changes alongside the prompts regardless of who made them.

The axis is deliberately honest about what it cannot establish: the chain is tamper-evident, not provably complete. Nothing can prove every prompt was recorded, so coverage intervals make the monitored windows explicit and gaps render as gaps.

The axis exists at the L1–L3 ceiling because L4 needs it. Without a transcript recorded before L1, the traceability rung is permanently out of reach for that project, so the evidence is collected whether or not the researcher intends to climb.

Scope and non-goals

Vaibify is:

  • A harness for containerizing a scientific workflow, recording its state in a content-addressed way, and verifying it against external authorities.

  • An implementation of the reproducibility ladder for a working scientist’s daily use.

  • A bridge between existing infrastructure (git, Zenodo, Overleaf, Docker) and the reproducibility claims a working scientist wants to make about their own work.

Vaibify is not:

  • A workflow management system. Nextflow, Snakemake, CWL, and Galaxy exist and are better at workflow orchestration. Vaibify uses a minimal JSON pipeline description because the target user is writing Python or shell scripts, not DSLs.

  • An AI agent. Vaibify does not generate code, reason about experiments, or draft papers. Systems like Denario, CMBAgent, and Sakana’s AI Scientist do that. Vaibify is the verification harness their output should pass through.

  • A hosted reproducibility platform. Whole Tale, Renku, and Code Ocean exist and serve users who want cloud-first workflows. Vaibify is local-first by design.

  • A cryptographic attestation service. Sigstore, Rekor, and in-toto exist for that. Vaibify integrates with them (or will, at L6); it does not replace them.

The larger bet

The scientific community’s trust infrastructure was built on the implicit assumption of human authorship: tenure, journal prestige, institutional affiliation, citation graphs. That infrastructure scales roughly linearly with human reviewers. Agentic research threatens to produce work at rates where reputation-based filtering is structurally unable to keep up.

The bet underlying vaibify is that the locus of scientific credibility will shift, over the next decade, from human expertise to machine-verifiable reproducibility — not replacing peer review, but supplementing it with a verifiable substrate that a reviewer (human or agent) can check in seconds rather than days. Getting astrophysics to L2 in 2026 is a small step toward that substrate. Getting the field to L6 by the time a remote-sensing result needs to be trusted as evidence for life beyond Earth is the ambition this framework anchors.

Vaibify’s contribution is intentionally modest: a working harness that one scientist can use today, shaped by the graded framework above, built to compose with the larger trust infrastructure as it emerges.