---
schema_version: 1
revision: "22ce5d34dbcb5a78"
id: "factory-preference-capture"
title: "Turn preference into reusable decisions"
---

# Turn preference into reusable decisions

Pairwise comparisons, honest rejection, recorded axes, and durable taste rules instead of vague adjectives.

## Related working material

- [Turn preference into reusable decisions — prompts and exercises](/topics/factory-preference-capture-practice/)
- [Turn preference into reusable decisions — questions and tradeoffs](/topics/factory-preference-capture-questions/)
- [Turn preference into reusable decisions — worked example](/topics/factory-preference-capture-worked-example/)

## TL;DR

- A local **this-or-that lab**: options render side by side, you arrow-key through Elo-paired duels, and every vote + note lands in `votes.jsonl` — data your agent reads, not vibes it guesses.
- You walk away with the whole kit (one harness page, one 30-line server, one manifest) and the loop: **generate options → vote → agent reads votes → sharper round**.
- The one move: never answer "what do you like?" — demand options and answer **"this or that"** instead. This workshop's own page template went from "ugly as sin" to shipped in 3 rounds and 130 votes.

 Have your agent set this up — paste this into it:

``` text
Fetch https://gentle-lasso-9d5n.here.now/agent.md?raw and follow it to set up a this-or-that voting lab in my current project. Run the success check at the end and show me the result.
```

## The principle

## Taste is a dataset, not a description

Ask someone what they like and you get adjectives — "clean", "modern", "not too busy" — words an agent can't execute. Show the same person two options and they answer instantly, reliably, hundreds of times in a row. That asymmetry is the whole trick. It's the same one RLHF is built on: humans are bad at writing reward functions and good at picking between two outputs. A this-or-that lab makes you the reward model, and `votes.jsonl` is the training signal your agent reads.

## The case study: 130 votes to a shipped template

This workshop's own pages are the receipt. The starting point was a template the author called "ugly as sin... extremely difficult to read at a glance." Round 1: eight wildly different page themes, king-of-the-hill voting. It produced a ranking in 24 seconds — and felt thin, because each loser died after one duel. Round 2 fixed the harness because the votes said so: Elo pairing so every option faces many opponents, a live leaderboard, both-bad and skip buttons, synced scrolling. Twelve themes, 88 votes, a winner shipped to ten live sites.

Then came the sharpest lesson. A refinement round built twelve variations orbiting the winner — and got rejected on sight: *"These designs are all WAAAY too similar."* Single-knob deltas waste votes; a duel only teaches something when the options genuinely diverge. The rebuilt round used structural moves — different section chrome, different code treatments, different layouts — plus a light/dark axis where duels alternated modes and the harness tracked per-mode records. Thirty votes later the data showed something no one would have articulated: one theme was undefeated in *both* modes (it shipped), another was a dark-mode specialist (4-0 dark, lost in light), a third the exact mirror. Total: 130 votes across three rounds, each round's harness and roster rebuilt from what the previous round's votes revealed.

## The rules that make votes cheap and honest

**Options must differ structurally.** If two options could be confused at arm's length, you're spending votes on noise. **Hold the subject constant** — vary only the approach, or the comparison teaches nothing (the same rule as the 9x image-mockup pattern in the Variations section). **Never force a choice:** both-bad is a real signal (it says the whole region of design space is wrong), and quitting early must be free — votes stop being honest the moment they feel like a chore. **Record every axis you care about** (mode, device width, section) on the vote itself; splits you didn't record are insights you lost. **Elo over knockout:** pairing least-played, closest-rated options keeps every duel informative, and ratings survive an unfinished session. And **notes beat scores** — a one-line "C's header wins but B's spacing is calmer" is worth ten silent votes to the agent reading the file.

## Beyond design: any decision an agent can render

Nothing in the kit knows about CSS. An option is any file an iframe can show — so the same loop ranks error-message copy, README structures, CLI output formats, API response shapes, function names, landing-page headlines, chart styles. The agent's job is always the same three steps: render genuinely distinct options, stay out of the way while you vote, then read `votes.jsonl` and turn your picks into rules it applies — and into a `DECISIONS.md` the next session inherits. That last step is the compounding one: taste captured once stops being re-litigated forever.

## Appendix

### Limits & caveats

- Small vote counts are directional, not statistical. 30 votes over 12 options ≈ 5 games each; Elo with K=32 at that depth is a rough sort. Treat standings as "top cluster / bottom cluster", not a precise order.
- The pairing heuristic (least-played, closest-rated, no immediate repeats) is not true adaptive sampling; informative pairs can be missed. In the case study, the \#1 and \#2 finishers never met head-to-head — read the pairings before over-trusting a margin.
- One voter = a personal taste model, not user research. It optimizes for *your* preference, which is the point — but don't present the output as market evidence.
- Iframe rendering means options should be same-origin and self-contained; cross-origin options lose scroll-sync and the mode toggle (both fail silently by design).
- The light/dark axis requires options to respond to `<html data-mode="...">`; options styled with only `prefers-color-scheme` won't flip inside the harness.
- "Both bad" applies a flat −12 to both options — a pragmatic penalty, not principled Elo.

## Original references

- [Turn preference into reusable decisions](https://gentle-lasso-9d5n.here.now/)
