Pipelines

A pipeline defines a sequence of steps to execute inside the Vaibify. Steps are self-contained units of work – each one runs a series of commands in order and produces output files such as figures or data products.

Project File

Pipelines are defined in a project.json under .vaibify/projects/ in the project repository. That is where the dashboard and vaibify run look for it; vaibify init --template writes it there. (.vaibify/ workflows/ is the legacy location and is still read, so existing repositories keep working.)

The file has four top-level fields:

Field	Type	Description
`sPlotDirectory`	string	Directory where figures are collected
`sFigureType`	string	Default figure format (`pdf`, `png`)
`iNumberOfCores`	integer	Cores to use (`-1` = all minus one)
`listSteps`	array	Ordered list of step objects

Step Object

Each step in listSteps has the following required fields:

Field	Type	Description
`sName`	string	Unique step identifier
`sDirectory`	string	Working directory for the step
`saPlotCommands`	string array	Shell commands to execute in order
`saPlotFiles`	string array	Output file paths produced

And these optional fields:

Field	Type	Default	Description
`bRunEnabled`	boolean	`true`	Whether the step is included in runs (run scope; verification iterates every step regardless)
`bPlotOnly`	boolean	`true`	Step produces only plots
`bInteractive`	boolean	`false`	Pause pipeline for user input
`saDataCommands`	string array	`[]`	Commands to run before plots
`saOutputDataFiles`	string array	`[]`	Output data files to verify
`saTestCommands`	string array	`[]`	Pytest commands for the step

Project size limits

Vaibify shows a one-shot “Project milestone” modal the first time a project’s listSteps reaches 100 entries. The acknowledgment is persisted in the repository’s .vaibify/state.json as bWarnedHundredSteps, so the warning does not reappear on reload or on subsequent additions. The threshold exists because polling cost (file-status, repos, discovery) grows roughly linearly with the number of tracked outputs, scripts, and markers, and beyond about 100 steps users typically notice some latency in the dashboard.

Vaibify refuses to add a 501st step to any project. The dashboard shows a “Step limit reached” modal; the backend rejects direct API calls with HTTP 400. The rationale is that the per-poll Docker exec budget and the dashboard render budget both break down beyond this scale. If a project requires more than 500 steps, split it into sibling projects within the same repository — vaibify supports multiple projects per container.

Example

{
    "sPlotDirectory": "Plot",
    "sFigureType": "pdf",
    "iNumberOfCores": -1,
    "listSteps": [
        {
            "sName": "RunSimulation",
            "sDirectory": "examples/EarthWater",
            "bRunEnabled": true,
            "bPlotOnly": false,
            "saDataCommands": [
                "python runAnalysis.py"
            ],
            "saPlotCommands": [
                "python makePlot.py"
            ],
            "saPlotFiles": []
        },
        {
            "sName": "PlotResults",
            "sDirectory": "examples/EarthWater",
            "bRunEnabled": true,
            "bPlotOnly": true,
            "saDataCommands": [],
            "saPlotCommands": [
                "python makePlot.py"
            ],
            "saPlotFiles": []
        }
    ]
}

Core Allocation

When iNumberOfCores is set to -1, the pipeline runner detects the total number of available cores and uses all but one. This leaves one core free for the operating system and other processes. Specify a positive integer to fix the core count explicitly.

Running a Pipeline

Start the container and execute the pipeline:

vaibify start

The pipeline runs all steps in order. Each step runs its commands sequentially. Steps themselves execute one at a time by default; future versions may support parallel step execution for independent steps.

What a long step survives

Steps in this project take hours to months, so it matters which failures reach them. Measured against a live daemon rather than assumed:

Closing the browser does not stop a run. The loop runs server-side and nothing cancels it on disconnect. Even a session that reaches its lifetime cap is orphaned, not released: the container keeps its lock and its work, and a fresh tab (vaibify open) reconciles with the run already in progress.
Restarting the dashboard server does not stop a run either. An in-container process outlives the process that launched it. What is lost is the live output stream — the restarted server can still learn that the step finished and with what exit code, but not the lines it was not there to read.

The practical rule for a step you intend to leave running: write what matters to a file. Anything that exists only in the streamed log is recoverable only while somebody is watching. This is also why every cross-step reference is a declared token rather than a hidden path — the dependency graph has to be readable from the project file alone, without replaying a run.

Pipeline Output

Figures produced by step commands are copied to the sPlotDirectory after each step completes. The directory is created automatically if it does not exist.

Integration with GitHub Actions

Warning

Not implemented. vaibify publish workflow — which would generate a GitHub Actions workflow from project.json — is not registered on the CLI, and the generator behind it has no caller anywhere in the product. Write the workflow by hand for now. See the Publishing section of the CLI reference for what exists today.

Multi-Container Projects

Each Vaibify project gets its own Docker image, container, and workspace volume. Multiple projects can run simultaneously on the same machine without interference. Use the --project/-p flag to target a specific project from any directory:

vaibify build -p earth-water
vaibify start -p earth-water
vaibify status -p earth-water
vaibify stop -p earth-water

Projects are registered automatically when you run vaibify init. When only one project is registered, the --project flag can be omitted. When you are inside a project directory (one containing vaibify.yml), the flag defaults to that project.

Host projects

A pipeline can also run directly on your machine (a host project, chosen when the project is added). The workflow file, the step contract, the tokens, and the dashboard are identical — the project file is portable between the two modes — but three differences matter:

Your environment is the environment. Steps run with your own user and whatever tools your machine has; there is no image build and no isolation. Vaibify shows a warning when you enter a host project, and every host terminal session opens with a reminder that processes you start can outlive the session.
Release proves less. Stopping a container proves everything in it stopped. A host project’s release proves the weaker claim that every process vaibify started has exited — and when a terminal was used, vaibify reports quiescence as unproven and routes you to vaibify reconcile rather than claiming quiet it cannot prove.
The PROOF ladder tops out early. Level 3 and Supervised mode certify properties only a container carries; a host project shows the containerization step as its remediation.

Host mode is for experimentation and first contact. When the analysis becomes real, create a container project and open the same repository there.