{
  "schemaVersion": "1",
  "revision": "22ce5d34dbcb5a78",
  "id": "factory-preference-capture",
  "slug": "preference-capture",
  "title": "Turn preference into reusable decisions",
  "summary": "Pairwise comparisons, honest rejection, recorded axes, and durable taste rules instead of vague adjectives.",
  "category": "Planning",
  "tags": [
    "preference",
    "factory"
  ],
  "sources": [
    {
      "label": "Turn preference into reusable decisions",
      "url": "https://gentle-lasso-9d5n.here.now/"
    }
  ],
  "sourceUnits": [
    "10-this-or-that/index.html#TL;DR",
    "10-this-or-that/index.html#The principle",
    "10-this-or-that/index.html#Appendix",
    "10-this-or-that/index.html#Taste is a dataset, not a description",
    "10-this-or-that/index.html#The case study: 130 votes to a shipped template",
    "10-this-or-that/index.html#The rules that make votes cheap and honest",
    "10-this-or-that/index.html#Beyond design: any decision an agent can render"
  ],
  "collection": "factory",
  "categoryId": "planning",
  "wordCount": 948,
  "url": "/topics/factory-preference-capture/",
  "markdownUrl": "/markdown/factory-preference-capture.md?raw=1&v=22ce5d34dbcb5a78",
  "jsonUrl": "/api/v1/documents/factory-preference-capture.json",
  "headings": [
    {
      "id": "related-working-material",
      "text": "Related working material",
      "level": 2
    },
    {
      "id": "tl-dr",
      "text": "TL;DR",
      "level": 2
    },
    {
      "id": "the-principle",
      "text": "The principle",
      "level": 2
    },
    {
      "id": "taste-is-a-dataset-not-a-description",
      "text": "Taste is a dataset, not a description",
      "level": 2
    },
    {
      "id": "the-case-study-130-votes-to-a-shipped-template",
      "text": "The case study: 130 votes to a shipped template",
      "level": 2
    },
    {
      "id": "the-rules-that-make-votes-cheap-and-honest",
      "text": "The rules that make votes cheap and honest",
      "level": 2
    },
    {
      "id": "beyond-design-any-decision-an-agent-can-render",
      "text": "Beyond design: any decision an agent can render",
      "level": 2
    },
    {
      "id": "appendix",
      "text": "Appendix",
      "level": 2
    },
    {
      "id": "limits-caveats",
      "text": "Limits & caveats",
      "level": 3
    }
  ],
  "markdown": "# Turn preference into reusable decisions\n\nPairwise comparisons, honest rejection, recorded axes, and durable taste rules instead of vague adjectives.\n\n## Related working material\n\n- [Turn preference into reusable decisions — prompts and exercises](/topics/factory-preference-capture-practice/)\n- [Turn preference into reusable decisions — questions and tradeoffs](/topics/factory-preference-capture-questions/)\n- [Turn preference into reusable decisions — worked example](/topics/factory-preference-capture-worked-example/)\n\n## TL;DR\n\n- A local **this-or-that lab**: options render side by side, you arrow-key through Elo-paired duels, and every vote + note lands in `votes.jsonl` — data your agent reads, not vibes it guesses.\n- You walk away with the whole kit (one harness page, one 30-line server, one manifest) and the loop: **generate options → vote → agent reads votes → sharper round**.\n- The one move: never answer \"what do you like?\" — demand options and answer **\"this or that\"** instead. This workshop's own page template went from \"ugly as sin\" to shipped in 3 rounds and 130 votes.\n\n Have your agent set this up — paste this into it:\n\n``` text\nFetch https://gentle-lasso-9d5n.here.now/agent.md?raw and follow it to set up a this-or-that voting lab in my current project. Run the success check at the end and show me the result.\n```\n\n## The principle\n\n## Taste is a dataset, not a description\n\nAsk someone what they like and you get adjectives — \"clean\", \"modern\", \"not too busy\" — words an agent can't execute. Show the same person two options and they answer instantly, reliably, hundreds of times in a row. That asymmetry is the whole trick. It's the same one RLHF is built on: humans are bad at writing reward functions and good at picking between two outputs. A this-or-that lab makes you the reward model, and `votes.jsonl` is the training signal your agent reads.\n\n## The case study: 130 votes to a shipped template\n\nThis workshop's own pages are the receipt. The starting point was a template the author called \"ugly as sin... extremely difficult to read at a glance.\" Round 1: eight wildly different page themes, king-of-the-hill voting. It produced a ranking in 24 seconds — and felt thin, because each loser died after one duel. Round 2 fixed the harness because the votes said so: Elo pairing so every option faces many opponents, a live leaderboard, both-bad and skip buttons, synced scrolling. Twelve themes, 88 votes, a winner shipped to ten live sites.\n\nThen came the sharpest lesson. A refinement round built twelve variations orbiting the winner — and got rejected on sight: *\"These designs are all WAAAY too similar.\"* Single-knob deltas waste votes; a duel only teaches something when the options genuinely diverge. The rebuilt round used structural moves — different section chrome, different code treatments, different layouts — plus a light/dark axis where duels alternated modes and the harness tracked per-mode records. Thirty votes later the data showed something no one would have articulated: one theme was undefeated in *both* modes (it shipped), another was a dark-mode specialist (4-0 dark, lost in light), a third the exact mirror. Total: 130 votes across three rounds, each round's harness and roster rebuilt from what the previous round's votes revealed.\n\n## The rules that make votes cheap and honest\n\n**Options must differ structurally.** If two options could be confused at arm's length, you're spending votes on noise. **Hold the subject constant** — vary only the approach, or the comparison teaches nothing (the same rule as the 9x image-mockup pattern in the Variations section). **Never force a choice:** both-bad is a real signal (it says the whole region of design space is wrong), and quitting early must be free — votes stop being honest the moment they feel like a chore. **Record every axis you care about** (mode, device width, section) on the vote itself; splits you didn't record are insights you lost. **Elo over knockout:** pairing least-played, closest-rated options keeps every duel informative, and ratings survive an unfinished session. And **notes beat scores** — a one-line \"C's header wins but B's spacing is calmer\" is worth ten silent votes to the agent reading the file.\n\n## Beyond design: any decision an agent can render\n\nNothing in the kit knows about CSS. An option is any file an iframe can show — so the same loop ranks error-message copy, README structures, CLI output formats, API response shapes, function names, landing-page headlines, chart styles. The agent's job is always the same three steps: render genuinely distinct options, stay out of the way while you vote, then read `votes.jsonl` and turn your picks into rules it applies — and into a `DECISIONS.md` the next session inherits. That last step is the compounding one: taste captured once stops being re-litigated forever.\n\n## Appendix\n\n### Limits & caveats\n\n- Small vote counts are directional, not statistical. 30 votes over 12 options ≈ 5 games each; Elo with K=32 at that depth is a rough sort. Treat standings as \"top cluster / bottom cluster\", not a precise order.\n- The pairing heuristic (least-played, closest-rated, no immediate repeats) is not true adaptive sampling; informative pairs can be missed. In the case study, the \\#1 and \\#2 finishers never met head-to-head — read the pairings before over-trusting a margin.\n- One voter = a personal taste model, not user research. It optimizes for *your* preference, which is the point — but don't present the output as market evidence.\n- Iframe rendering means options should be same-origin and self-contained; cross-origin options lose scroll-sync and the mode toggle (both fail silently by design).\n- The light/dark axis requires options to respond to `<html data-mode=\"...\">`; options styled with only `prefers-color-scheme` won't flip inside the harness.\n- \"Both bad\" applies a flat −12 to both options — a pragmatic penalty, not principled Elo."
}
