Controlled variation workflows: practical reference

Controlled software-variation workflow covering setup-only agent prompts, single-variable matrices, baseline repository capture, isolation and fan-out, identical verification, diff and UI review, five decision outcomes, convergence, troubleshooting, and…

Agent Setup Prompt

Set up a controlled variation run before any candidate implementation starts. Build the files, folders, isolation lanes, verification ledger, review surfaces, and saved-decision template first, then stop and report whether setup is ready.

1. Work from the repository root. Choose a short run id that names the software decision, such as `billing-webhook-$(date +%Y%m%d-%H%M%S)`, and set `ROOT="artifacts/variations/$RUN_ID"`.

2. Create the run structure: `prompts/`, `launch/`, `logs/`, `review/`, `decisions/`, and `candidates/v1`, `candidates/v2`, `candidates/v3` under `$ROOT`. If the variation is visual, also create `review/screenshots/`.

3. Write `$ROOT/matrix.md` before launching anything. Include Goal, Invariants, Variable Under Test, Candidate Count, Candidates, and Review Criteria. The matrix must name exactly one variable. Keep repo state, base prompt, target feature or bug, constraints, verification command, and review criteria identical for every candidate. If a second dimension matters, record it as a future round.

4. Save the shared base prompt to `$ROOT/prompts/base.prompt.md`. Save matching launch metadata to `$ROOT/launch/base.launch.json` using Prompt Builder when available, for example `p --print-prompt "$PROMPT"` and `p --print-launch-json "$PROMPT"`. If Prompt Builder is unavailable, still write the base prompt file and create a small valid launch JSON file that records the prompt text, run id, candidate ids, artifact root, isolation method, and verification command.

5. Capture baseline repo state before fan-out: save `git status --short` to `$ROOT/review/git-status-before.txt`, `git diff` to `$ROOT/review/dirty-before.patch`, and `git diff --staged` to `$ROOT/review/staged-before.patch`. If the repo has unknown dirty state, stop and report it instead of creating candidate code lanes.

6. Choose the isolation strategy and record it. For prompt, plan, or docs-only variations, use `$ROOT/candidates/v1`, `v2`, and `v3`. For parallel code edits, prefer git worktrees at `../variation-$RUN_ID/v1`, `v2`, and `v3`, each on its own branch, then save `git worktree list` to `$ROOT/review/worktrees.txt`. Each candidate must have a distinct branch, worktree, or directory.

7. Create `$ROOT/prompts/v1.prompt.md`, `$ROOT/prompts/v2.prompt.md`, and `$ROOT/prompts/v3.prompt.md` from the same base prompt. Append only the candidate override: candidate id, variable name, that candidate's setting, and the instruction to change no other dimension.

8. Define one verification command and save it in the matrix plus `$ROOT/review/verification-command.txt`. Create `$ROOT/review/verification.tsv` with columns for candidate id, status, and command. Do not run implementation verification yet. The command must be identical for every candidate; if a candidate needs different proof, make that a new variation round.

9. Create review artifacts before generation: `$ROOT/review/comparison.md` with Goal, Variable Under Test, Verification, Diff Review, Review Criteria, and Notes sections; placeholder paths for `logs/v1.verify.log`, `logs/v2.verify.log`, `logs/v3.verify.log`; placeholder paths for `review/v1.diffstat.txt`, `review/v1.diff.patch`, and matching v2/v3 files. Leave results blank until candidates exist.

10. Create `$ROOT/decisions/selection.md` with fields for outcome, primary candidate, secondary candidate ids, decision, why, rejected candidates, required follow-up, final verification command, per-candidate notes, and a continue prompt. The outcome must later be exactly one of Pick, Combine, Refine, Expand, or Reject.

11. Run setup checks only: confirm the matrix, base prompt, launch JSON, candidate prompts, repo-state files, verification command file, comparison template, decision template, and isolation lanes exist. Print OK or FAIL for each check and show the run file list with `find "$ROOT" -maxdepth 3 -type f | sort`.

12. Respect privacy boundaries. Do not copy secrets, tokens, cookies, private endpoints, full environment dumps, raw private config, or unrelated home-directory files into prompts, logs, launch JSON, matrix files, review artifacts, or decisions. Summarize sensitive configuration instead of pasting it.

13. Report back with the artifact root, candidate ids, chosen isolation strategy, single variable under test, single verification command, repo-state classification, setup check results, any files you intentionally did not copy for privacy, and any blocker. Do not generate candidate implementations, run candidate verification, fill comparison results, or make a Pick/Combine/Refine/Expand/Reject decision until setup verification passes and the next step is explicitly requested.

Big Idea

Phase: Generation And Evidence - use the prepared surface to generate comparable options and preserve the proof that decides between them.

Code is free does not mean quality is free.

Try This Now

Finish this sentence before generating candidates: We are comparing candidates that differ only by __________. For a billing webhook bug, try: idempotency strategy. Write the same verification command before any fan-out.

It means another candidate is cheap enough that the workflow should change:

do not bet everything on the first generated answer

Produce candidates, verify them, compare them, converge deliberately, and save the decision.

Cheap:

  • another implementation option
  • another UI integration strategy
  • another idempotency strategy
  • another test boundary
  • another prompt framing
  • another documentation organization

Still expensive:

  • understanding the problem
  • keeping candidates controlled
  • integrating the winner
  • running verification
  • resolving conflicts
  • preserving why the decision was made

Practical rule:

Generate more because comparison improves the decision, not because you are avoiding the decision.

When To Reach For This

Use variations when several plausible approaches exist and comparison will make the final choice better.

Use Variations For Example
UI integration strategy saved payment methods in checkout: inline selector vs drawer vs review-step picker
idempotency strategy billing webhook duplicate-processing: provider event guard vs invoice lock vs processing ledger
auth/session architecture middleware-first vs session-store-first vs token-rotation boundary
state management local reducer vs server state cache vs URL-backed state
test boundary unit regression vs integration replay vs end-to-end checkout path
API shape command endpoint vs resource endpoint vs explicit workflow endpoint
data loading model route loader vs component query vs server action
implementation alternatives minimal patch vs extracted helper vs explicit state machine
prompt framing plan-first vs patch-first vs test-first
documentation organization quickstart-first vs concept-first vs task checklist

Skip variations when another candidate adds noise.

Skip Variations For Better Move
one-line typo fix patch it and test it
obvious hotfix apply the smallest safe change
strict migration follow the migration plan exactly
exact requested implementation build what was specified
unverifiable candidates define verification first
dirty repo with no saved state checkpoint before fan-out
failing test names the exact bug fix the bug first
candidates would share unsafe state isolate before launching

Hard rule:

Variation is a decision tool, not a procrastination tool.

How This Used To Work

What you used to expect:

Ask for one implementation, inspect it, tweak it, and hope the first path was good enough.

What the workflow gives you now:

Start from one deliberate prompt, generate controlled candidates, verify each
the same way, compare results, then save the chosen path.

What you used to expect:

More candidates automatically means better work.

What the workflow gives you now:

Unverified candidates are just noise. The value comes from controlled
differences, identical verification, and a saved decision.

Core Pattern

goal
-> variable
-> matrix
-> isolation
-> fan-out
-> verify
-> review
-> selection
-> convergence
-> saved decision

Do not skip the middle:

fan-out without isolation = collisions
fan-out without verification = noise
fan-out without review = drift
fan-out without saved decision = repeated debate

One Dimension Rule

Vary exactly one dimension across candidates.

Finish this sentence before launching anything:

We are comparing candidates that differ only by __________.

Good variables:

Variable Good Comparison Question
UI integration strategy Where should saved payment methods enter checkout?
idempotency strategy Which duplicate-processing guard is safest for billing webhooks?
state management Which state boundary keeps the checkout flow simplest?
test boundary Which test layer catches this regression with the least brittleness?
API shape Which endpoint shape keeps auth/session changes clear?
data loading model Where should this screen load account and payment data?
prompt framing Which prompt produces the most verifiable implementation plan?
documentation organization Which doc structure helps the next maintainer act fastest?

Bad variables:

Bad Variable Why It Fails
random ideas no controlled comparison
each one different impossible to know what mattered
explore everything no review question
make five better versions vague and untestable
let every agent decide candidates drift apart

If two dimensions matter, run two rounds.

Round 1: compare idempotency strategy.
Round 2: compare test boundary for the selected strategy.

Variation Matrix

Write the matrix before launching candidates.

# Variation Matrix

## Goal

<What software outcome are we trying to achieve?>

## Invariants

These stay the same for every candidate:

- same repo state:
- same base prompt:
- same target feature or bug:
- same constraints:
- same verification command:
- same review criteria:

## Variable Under Test

- variable:
- why this matters:
- what is out of scope:

## Candidate Count

- count:
- ids: v1, v2, v3

## Candidates

| ID | Variable setting | Output location | Verification command | Notes |
| --- | --- | --- | --- | --- |
| v1 | | | | |
| v2 | | | | |
| v3 | | | | |

## Review Criteria

Compare candidates on:

1. passes verification
2. preserves invariants
3. smallest durable change
4. easiest integration path
5. clearest rollback path

Example: Billing Webhook Duplicate Processing

# Variation Matrix

## Goal

Prevent duplicate processing of paid billing webhook events.

## Invariants

- Provider webhook contract stays fixed.
- Database schema stays fixed for this round.
- Existing invoice state transitions stay compatible.
- Same verification command: `pnpm test billing-webhook -- --runInBand && pnpm build`.

## Variable Under Test

Idempotency strategy.

## Candidates

| ID | Variable setting | Output location | Verification command |
| --- | --- | --- | --- |
| v1 | provider event id guard | `../variation-<run-id>/v1` | `pnpm test billing-webhook -- --runInBand && pnpm build` |
| v2 | invoice-level processing lock | `../variation-<run-id>/v2` | `pnpm test billing-webhook -- --runInBand && pnpm build` |
| v3 | webhook processing ledger | `../variation-<run-id>/v3` | `pnpm test billing-webhook -- --runInBand && pnpm build` |

## Review Criteria

Which strategy prevents duplicate invoice processing with the smallest durable change?

Start From The Section 5 Seed

Use Prompt Builder to save the base prompt and launch JSON under the variation run.

RUN_ID="billing-webhook-$(date +%Y%m%d-%H%M%S)"
ROOT="artifacts/variations/$RUN_ID"
mkdir -p "$ROOT"/{prompts,launch,logs,review,decisions}
mkdir -p "$ROOT"/candidates/{v1,v2,v3}

PROMPT='Generate three controlled fixes for duplicate billing webhook processing. Keep the provider contract and database schema fixed. Vary only the idempotency strategy. Each option must name files, tests, risks, and rollback notes.'

p --print-prompt "$PROMPT" \
  > "$ROOT/prompts/base.prompt.md"

p --print-launch-json "$PROMPT" \
  > "$ROOT/launch/base.launch.json"

test -s "$ROOT/prompts/base.prompt.md" && echo "base-prompt:OK"
jq '.mode, .prompt' "$ROOT/launch/base.launch.json"

Save the matrix next to the seed:

cat > "$ROOT/matrix.md" <<'EOF'
# Variation Matrix

## Goal

Prevent duplicate processing of paid billing webhook events.

## Invariants

- Provider webhook contract stays fixed.
- Database schema stays fixed.
- Existing invoice state transitions stay compatible.
- Same verification command: `pnpm test billing-webhook -- --runInBand && pnpm build`.

## Variable Under Test

Idempotency strategy.

## Candidate Count

- count: 3
- ids: v1, v2, v3

## Candidates

| ID | Variable setting | Output location | Verification command | Notes |
| --- | --- | --- | --- | --- |
| v1 | provider event id guard | `../variation-<run-id>/v1` | `pnpm test billing-webhook -- --runInBand && pnpm build` | |
| v2 | invoice-level processing lock | `../variation-<run-id>/v2` | `pnpm test billing-webhook -- --runInBand && pnpm build` | |
| v3 | webhook processing ledger | `../variation-<run-id>/v3` | `pnpm test billing-webhook -- --runInBand && pnpm build` | |

## Review Criteria

- prevents duplicate processing
- preserves provider contract
- keeps rollback simple
- passes the same verification command
EOF

Confirm the setup:

find "$ROOT" -maxdepth 3 -type d | sort
test -s "$ROOT/matrix.md" && echo "matrix:OK"
test -s "$ROOT/prompts/base.prompt.md" && echo "base-prompt:OK"
test -s "$ROOT/launch/base.launch.json" && echo "launch-json:OK"

Save Repo State Before Fan-Out

Before parallel code edits, save the current state.

git status --short | tee "$ROOT/review/git-status-before.txt"
git diff > "$ROOT/review/dirty-before.patch"
git diff --staged > "$ROOT/review/staged-before.patch"

If the repo is dirty, choose one:

State Safe Next Move
intentional local work commit or stash before fan-out
unrelated local work move it out of the candidate path
generated files only clean or save them under artifacts/
unsure stop and inspect before launching candidates

Do not fan out code edits from unknown dirty state.

Choose Isolation

Parallel candidates must not overwrite each other.

Isolation Option Use When Example
copied prompt/output folders candidates are plans, docs, prompts, or generated artifacts artifacts/variations/<run-id>/candidates/v1
branch edits are small and reviewed one at a time git switch -c variation/<run-id>-v1
git worktree candidates need parallel repo edits git worktree add ../variation-<run-id>/v1 -b variation/<run-id>-v1
directory copy no git worktree support, but file edits must be separated cp -R app app-v1
feature flag candidates can coexist in one app behind flags CHECKOUT_VARIANT=v1
unique process state candidates run services PORT=3101, PORT=3102, PORT=3103

For code edits, prefer worktrees:

BASE="$(git rev-parse --show-toplevel)"
PARENT="$(dirname "$BASE")/variation-$RUN_ID"

git -C "$BASE" worktree add "$PARENT/v1" -b "variation/$RUN_ID-v1"
git -C "$BASE" worktree add "$PARENT/v2" -b "variation/$RUN_ID-v2"
git -C "$BASE" worktree add "$PARENT/v3" -b "variation/$RUN_ID-v3"

git -C "$BASE" worktree list | tee "$ROOT/review/worktrees.txt"

Build Candidate Prompts

Start with the same base prompt. Add only the candidate setting.

for id in v1 v2 v3; do
  cp "$ROOT/prompts/base.prompt.md" "$ROOT/prompts/$id.prompt.md"
done

cat >> "$ROOT/prompts/v1.prompt.md" <<'EOF'
Candidate override:
- candidate id: v1
- variable: idempotency strategy
- setting: provider event id guard
- do not change any other dimension
EOF

cat >> "$ROOT/prompts/v2.prompt.md" <<'EOF'
Candidate override:
- candidate id: v2
- variable: idempotency strategy
- setting: invoice-level processing lock
- do not change any other dimension
EOF

cat >> "$ROOT/prompts/v3.prompt.md" <<'EOF'
Candidate override:
- candidate id: v3
- variable: idempotency strategy
- setting: webhook processing ledger
- do not change any other dimension
EOF

for id in v1 v2 v3; do
  test -s "$ROOT/prompts/$id.prompt.md" && echo "$id prompt:OK"
done

For checkout UI integration, the same shape works:

variable: UI integration strategy
v1 setting: inline payment method selector in checkout form
v2 setting: account drawer payment method selector
v3 setting: review-step payment method selector

Keep database contracts, provider APIs, security constraints, and verification fixed unless those are the chosen variable.

Fan-Out

Use the same launch shape for every candidate.

Local background shape:

GEN="codex"

for id in v1 v2 v3; do
  (
    cd "$PARENT/$id"
    "$GEN" "$(cat "$ROOT/prompts/$id.prompt.md")"
  ) > "$ROOT/logs/$id.generate.log" 2>&1 &
done

wait

Visible pane shape:

cmux-run-surface "cd '$PARENT/v1' && codex \"$(cat '$ROOT/prompts/v1.prompt.md')\""
cmux-run-surface "cd '$PARENT/v2' && codex \"$(cat '$ROOT/prompts/v2.prompt.md')\""
cmux-run-surface "cd '$PARENT/v3' && codex \"$(cat '$ROOT/prompts/v3.prompt.md')\""

What to preserve:

same base prompt
same invariants
same launch shape
one isolated lane per candidate
one log per candidate

Verify Every Candidate

Run the same verification command for every candidate.

VERIFY='pnpm test billing-webhook -- --runInBand && pnpm build'
: > "$ROOT/review/verification.tsv"

for id in v1 v2 v3; do
  (
    cd "$PARENT/$id"
    bash -lc "$VERIFY"
  ) > "$ROOT/logs/$id.verify.log" 2>&1
  status=$?
  printf '%s\t%s\t%s\n' "$id" "$status" "$VERIFY" \
    >> "$ROOT/review/verification.tsv"
done

column -t -s $'\t' "$ROOT/review/verification.tsv"

Save diff material for review:

for id in v1 v2 v3; do
  git -C "$PARENT/$id" diff --stat \
    > "$ROOT/review/$id.diffstat.txt"
  git -C "$PARENT/$id" diff \
    > "$ROOT/review/$id.diff.patch"
done

For UI candidates, also save screenshots:

mkdir -p "$ROOT/review/screenshots"
# Save:
# "$ROOT/review/screenshots/v1.png"
# "$ROOT/review/screenshots/v2.png"
# "$ROOT/review/screenshots/v3.png"

Review Surface

Use the lightest surface that makes comparison honest.

Surface Use When Save
simple matrix text, plans, test results, diffs review/comparison.md
diff review code changes matter most review/v*.diff.patch
screenshots UI integration or visual changes review/screenshots/
test results correctness is the main gate logs/*.verify.log, review/verification.tsv
structured feedback choice needs to be preserved exactly decisions/selection.md or saved JSON

Create comparison notes:

cat > "$ROOT/review/comparison.md" <<EOF
# Variation Comparison

## Goal

Prevent duplicate processing of paid billing webhook events.

## Variable Under Test

Idempotency strategy.

## Verification

| Candidate | Status | Log |
| --- | --- | --- |
| v1 | see \`verification.tsv\` | \`$ROOT/logs/v1.verify.log\` |
| v2 | see \`verification.tsv\` | \`$ROOT/logs/v2.verify.log\` |
| v3 | see \`verification.tsv\` | \`$ROOT/logs/v3.verify.log\` |

## Diff Review

| Candidate | Diff Stat | Full Diff |
| --- | --- | --- |
| v1 | \`$ROOT/review/v1.diffstat.txt\` | \`$ROOT/review/v1.diff.patch\` |
| v2 | \`$ROOT/review/v2.diffstat.txt\` | \`$ROOT/review/v2.diff.patch\` |
| v3 | \`$ROOT/review/v3.diffstat.txt\` | \`$ROOT/review/v3.diff.patch\` |

## Review Criteria

- prevents duplicate processing
- preserves provider contract
- keeps rollback simple
- passes the same verification command
- minimizes unnecessary code movement

## Notes

| Candidate | Strengths | Risks | Keep / Drop |
| --- | --- | --- | --- |
| v1 | | | |
| v2 | | | |
| v3 | | | |
EOF

Decision Outcomes

End every review with one action.

Outcome Meaning Next Step
Pick choose one candidate as-is polish and verify selected candidate
Combine use one primary candidate plus named pieces from others merge only named pieces, then verify
Refine keep one candidate and apply specific feedback apply bounded edits, then verify
Expand run another controlled round on a new dimension write a new matrix
Reject throw away the set rewrite the prompt, matrix, or verification

Saved decision shape:

cat > "$ROOT/decisions/selection.md" <<'EOF'
# Saved Decision

- outcome:
- primary candidate:
- secondary candidate ids:
- decision:
- why:
- rejected candidates:
- required follow-up:
- final verification command:

## Review Notes

- v1:
- v2:
- v3:

## Continue Prompt

Continue from this saved decision.
Do not reopen the entire search.
Preserve the selected candidate.
Apply only the named feedback.
Run final verification and save the result.
EOF

Use the saved decision as the next prompt material:

cat "$ROOT/decisions/selection.md" \
  | p --stdin --print-prompt \
  "Use this saved decision to converge the variation set. Keep scope narrow and run the required verification." \
  > "$ROOT/prompts/converge.prompt.md"

test -s "$ROOT/prompts/converge.prompt.md" && echo "converge-prompt:OK"

Ask The Agent This

Create a variation matrix:

Create a controlled variation matrix for adding saved payment methods to checkout.

Use exactly one variable: UI integration strategy. Keep database schema,
provider API, auth behavior, and verification fixed. Propose three candidates,
the same verification command for each, isolation strategy, review criteria,
and the saved decision file shape.

Generate candidate prompts:

Using this base Prompt Builder prompt, create three candidate prompts.

Each candidate must preserve the same goal, invariants, verification command,
and review criteria. Only vary the idempotency strategy for the billing webhook:
provider event id guard, invoice-level processing lock, and webhook processing
ledger. Do not introduce a second variable.

Review a finished variation set:

Review this variation run.

Inspect the matrix, candidate prompts, verification logs, diffs, and comparison
notes. Decide Pick, Combine, Refine, Expand, or Reject. If Combine, choose one
primary candidate and name the exact pieces to take from the others.

Converge after selection:

Use this saved decision to converge the variation set.

Do not reopen the entire search. Preserve the selected candidate, apply only
the named feedback, run the final verification command, and save the result.

What To Confirm / How To Know It Worked

Goal Confirmation
base prompt saved test -s "$ROOT/prompts/base.prompt.md"
launch shape saved test -s "$ROOT/launch/base.launch.json"
matrix exists test -s "$ROOT/matrix.md"
one variable named matrix has one Variable Under Test
candidates isolated each candidate has a distinct branch/worktree/directory
same verification verification.tsv shows the same command for every candidate
logs saved `find "$ROOT/logs" -type f
diff/review material saved `find "$ROOT/review" -type f
comparison complete test -s "$ROOT/review/comparison.md"
decision saved test -s "$ROOT/decisions/selection.md"
convergence prompt saved test -s "$ROOT/prompts/converge.prompt.md"

If It Goes Wrong

Candidates Changed Multiple Variables

Stop and rewrite the matrix.

If the candidate changes UI, database schema, and test strategy, it is not a controlled variation.

Recovery:

Keep one chosen variable.
Move every other change into invariants or out-of-scope.
Regenerate candidate prompts.

Verification Differs Per Candidate

The comparison is weak if every candidate gets a different proof command.

Recovery:

Pick one verification command before generation.
If a candidate needs a different proof, that is evidence for a new round.

Repo Was Dirty Before Fan-Out

Dirty state is not forbidden. Unknown dirty state is the problem.

Recovery:

git status --short
git diff
git diff --staged

Then commit, stash, clean, or explicitly record what should carry into every candidate.

Best Idea Is Split Across Candidates

Use Combine, not vague merging.

primary candidate: v2
take from v1: rollback guard
take from v3: test fixture
forbidden: unrelated renames

Then run final verification after convergence.

All Candidates Fail

Do not pretend one failed candidate won.

Use:

Reject

or:

Expand with a narrower matrix

Save what the failures proved before starting the next round.

References And Resources

  • Prompt Builder seed prompt: p --print-prompt
  • Prompt Builder launch JSON: p --print-launch-json
  • Local run folder: artifacts/variations/<run-id>/
  • Candidate isolation: git worktree
  • Repo state checks: git status --short, git diff, git diff --staged
  • Review artifacts: review/verification.tsv, review/comparison.md, decisions/selection.md

Operational boundaries

  • The fan-out snippets describe the launch shape, not a universally executable harness. Resolve $ROOT to an absolute artifact directory before changing into worktrees; validate nested quoting and the installed Codex non-interactive invocation.
  • A candidate process exiting or wait returning is not passing proof. Capture each generation exit status and run the same project-valid verification command in every lane.
  • Git worktrees start from committed state. A plain git diff omits staged, untracked, and already committed candidate changes; preserve the actual baseline and all relevant review evidence.

Source references