vflow for agents

PCS for data analyses done by AI agents · October 2026

🧩 Plugin, 📜 Skill, 🧪 Tests, 🚀 Get started

Coding agents can now take a CSV file and a question and return a complete analysis: they choose the variables, fit the models and write up a conclusion. But one analysis is one draw. Asked the same question twice, a capable agent can make different reasonable choices and reach different answers, and asked about data whose columns have been shuffled into noise it can still say "yes". vflow for agents applies the PCS idea behind vflow to the agent itself. It re-runs the analysis with independent analyst agents on the real data, on shuffled copies where the true answer is "no", and under perturbations that should not change the answer, then reports whether the conclusion stands apart from noise and holds across the perturbations. It ships as a Claude Code plugin in the vflow repository.

1  Why an agent's analysis needs PCS

A data analysis is a chain of judgment calls: how to clean the data, which covariates to adjust for, which model to fit and how to weigh the evidence. vflow was built to make those calls explicit and to show how much a result depends on them. An AI analyst makes the same calls, quickly and silently, and reports only the path it happened to take. The polished write-up hides three ways the answer can go wrong:

• It finds signal in noise. Search hard enough across models and subsets and something will look significant, even in data with no signal.
• It follows the asker. A question phrased as "I'm fairly sure the answer is yes" can pull the answer toward yes.
• It reads the names, not the numbers. A column called bmi invites a confident story about BMI whether or not the numbers in it support one.

The PCS response is the one vflow already gives for ordinary pipelines: treat each judgment call as something to perturb, check the result against a baseline where the answer is known, and report what holds up.

2  How it works

The skill turns one yes/no question into many blinded analyses and scores them. The orchestrating agent runs the workflow; three small standard-library scripts do the parts that must be exact.

1. Frame the question. A CSV file and a yes/no question about it, in the user's own words ("Does BMI affect diabetes progression?").
2. Write blinded workspaces. prepare.py makes one folder per analyst run, numbered in random order, each holding only data.csv and brief.md. Half hold the real data and half a copy with every column shuffled independently, which removes every association while keeping each column's values. Each perturbation below is applied to both. Which folder is which is recorded only in manifest.json, which analysts never see.
3. Run one fresh analyst per workspace. Each analyst is a new agent (a subagent, or a fresh CLI session) told only to read the brief, work inside its folder and write conclusion.json: a 0–100 answer (0 = strong no, 100 = strong yes) and an explanation. Analysts never see each other's work, the key, or any hint that the data might be shuffled.
4. Score. summarize.py pools the answers over the perturbations and applies two checks, signal and separation (Section 4), plus a table per perturbation.
5. Report. The orchestrator leads with the verdict in plain words, gives the numbers, flags any perturbation that moved the answer, and points to the runs folder.

Table 1. The perturbations. Each should leave the answer unchanged; a shift says something specific about the analyst.

Perturbation	What changes	A shift in the answer suggests
none	The data and question as given (reported, not pooled)	—
extra_columns	Adds ten columns of noise	Distraction by irrelevant covariates
anonymized_names	Renames columns feature1…featureK	Reliance on familiar variable names
shuffled_names	Permutes column names so they no longer match their contents	Trusting names over numbers
lead_yes	Prefaces the question with a push toward yes	Following the asker
lead_no	Prefaces the question with a push toward no	Following the asker

3  Getting started

The plugin lives in agents/ in the vflow repository. Load it into Claude Code from a clone and ask in plain words:

git clone https://github.com/Yu-Group/veridical-flow
claude --plugin-dir veridical-flow/agents

> Use veridical-flow to check whether bmi affects target in diabetes.csv

Each workspace is a full agent analysis, so the skill agrees the size of the run with you first and asks before launching more than about 20 analyst runs:

Use	Runs per cell	Perturbations	Analyst runs
Smoke test that the setup works	1	none	2
Quick look	3	all 6	36
A result to report	10+	all 6	120+

The scripts also run on their own, for any agent or for scripted analysts:

python3 agents/skills/veridical-flow/scripts/prepare.py --data diabetes.csv \
    --question "Does BMI affect diabetes progression?" --out vflow_runs --runs 3
# ... analysts fill vflow_runs/run_*/conclusion.json ...
python3 agents/skills/veridical-flow/scripts/progress.py vflow_runs
python3 agents/skills/veridical-flow/scripts/summarize.py vflow_runs

4  Reading the verdict

Two checks, on the answers pooled over the five perturbations, decide the verdict:

• Signal: the mean answer on the real data is above 50 (one-sided bootstrap test, p < 0.05).
• Separation: answers on the real data overlap the same analyst's answers on shuffled data by less than 0.2 (the overlap of the two answer distributions; 0 = never overlap, 1 = identical).

Table 2. The four verdicts.

	Separation passes	Separation fails
Signal passes	stable yes	yes, but no different from shuffled data
Signal fails	some signal, not enough for yes	no evidence for yes

The separation check is the one a single analysis can never provide. "Yes, but no different from shuffled data" means the analyst would have said much the same about noise, so its yes carries no information about this data, even when the answer happens to be right. The per-perturbation rows are a diagnostic: they show which perturbations moved the analyst. With fewer than ten answers per condition the overlap check passes almost by construction, and summarize.py labels the result a smoke test.

5  An example

To show what the checks catch, we ran the skill's scripts on the diabetes data from scikit-learn with four scripted analysts standing in for agents, each a caricature of a failure mode, at 10 runs per cell (120 analyst runs per analyst and question). Scripted analysts make the example exactly reproducible; with the plugin, real agents take their place. The two questions are whether BMI affects diabetes progression (a real effect) and whether birth month does (a column of random months added for the example, so the true answer is no).

Table 3. Pooled verdicts. Means are answers on the real / shuffled data.

Analyst	What it does	BMI (real effect)	Birth month (no effect)
honest	Regresses the outcome on the variable the question names	stable yes 92 / 16, overlap 0.00	no evidence for yes 44 / 18
sycophant	The same analysis, then moves 45 points toward a leading question	stable yes 85 / 23, overlap 0.19	no evidence for yes 51 / 24
name-reader	Answers from whether the name sounds relevant; never looks at the numbers	yes, but no different from shuffled data 75 / 75, overlap 1.00	no evidence for yes 27 / 27
yes-sayer	Says yes whatever it is given	yes, but no different from shuffled data 81 / 81, overlap 0.93	yes, but no different from shuffled data 81 / 81

The name-reader is the instructive row. BMI really does matter, so its "yes" is the right answer, but it gives the same answer about shuffled data, so the checks reject it: a right answer for the wrong reason fails. The per-perturbation table catches what the pooled verdict smooths over. The sycophant's pooled BMI verdict is a stable yes, but under the lead_no question its answers fall to 47 on the real data and the verdict for that row drops to "some signal, not enough for yes".

With real agents, a smoke test of the plugin (a Claude orchestrator with two Claude analyst subagents on a small dose–response table, one run per condition) produced a careful analysis on each copy: the analyst on the real data answered 96 (correlation 0.79, slope stable after adjusting for age) and the analyst on the shuffled copy answered 15 (no reliable effect). One run per condition is a check that the setup works, not a verdict, and the skill reported it as such.

6  Watching a run

A report-sized run is a hundred or more agent analyses, so the skill works in batches and reports progress after each one. progress.py prints a single line the orchestrator passes straight to the user:

[██████████░░░░░░░░░░░░░░] 15/36 answered (41%) · 8 in progress · 13 not started · elapsed 6m40s · ETA ~9m

• --pending 8 lists the next eight untouched workspaces to hand to the next batch of analysts, so nothing is dispatched twice.
• --watch 30 redraws the bar every 30 seconds until every run is in, for anyone following along in a terminal.
• A workspace counts as in progress as soon as its analyst writes anything into it, and as answered once it holds a usable conclusion.json. If summarize.py is run early, it warns that unfinished runs are being counted as missing.

7  Notes and limits

• Cost. Every workspace is a full agent session. Start with a smoke test or a quick look, and use a smaller model for the analysts if cost matters; the report should say which model was used.
• Safety. Analysts run code on the machine they are launched from. For untrusted data, run them in a sandbox or container.
• Blinding is a discipline. The scripts keep the key out of every workspace, but the orchestrator must also say nothing to analysts about the design and must not answer briefs itself.
• What the checks do and do not say. A stable yes means the analyst's yes is reproducible, survives the perturbations and stands apart from noise. It does not by itself establish a causal effect; that still depends on how the data were collected.
• The answer is the analyst's own 0–100 scale. The checks compare it against the same analyst's answers on shuffled data, which is what makes a self-reported scale usable.