Turn preference into reusable decisions

Pairwise comparisons, honest rejection, recorded axes, and durable taste rules instead of vague adjectives.

Pairwise comparisons, honest rejection, recorded axes, and durable taste rules instead of vague adjectives.

TL;DR

  • A local this-or-that lab: options render side by side, you arrow-key through Elo-paired duels, and every vote + note lands in votes.jsonl — data your agent reads, not vibes it guesses.
  • You walk away with the whole kit (one harness page, one 30-line server, one manifest) and the loop: generate options → vote → agent reads votes → sharper round.
  • The one move: never answer "what do you like?" — demand options and answer "this or that" instead. This workshop's own page template went from "ugly as sin" to shipped in 3 rounds and 130 votes.

Have your agent set this up — paste this into it:

Fetch https://gentle-lasso-9d5n.here.now/agent.md?raw and follow it to set up a this-or-that voting lab in my current project. Run the success check at the end and show me the result.

The principle

Taste is a dataset, not a description

Ask someone what they like and you get adjectives — "clean", "modern", "not too busy" — words an agent can't execute. Show the same person two options and they answer instantly, reliably, hundreds of times in a row. That asymmetry is the whole trick. It's the same one RLHF is built on: humans are bad at writing reward functions and good at picking between two outputs. A this-or-that lab makes you the reward model, and votes.jsonl is the training signal your agent reads.

The case study: 130 votes to a shipped template

This workshop's own pages are the receipt. The starting point was a template the author called "ugly as sin... extremely difficult to read at a glance." Round 1: eight wildly different page themes, king-of-the-hill voting. It produced a ranking in 24 seconds — and felt thin, because each loser died after one duel. Round 2 fixed the harness because the votes said so: Elo pairing so every option faces many opponents, a live leaderboard, both-bad and skip buttons, synced scrolling. Twelve themes, 88 votes, a winner shipped to ten live sites.

Then came the sharpest lesson. A refinement round built twelve variations orbiting the winner — and got rejected on sight: "These designs are all WAAAY too similar." Single-knob deltas waste votes; a duel only teaches something when the options genuinely diverge. The rebuilt round used structural moves — different section chrome, different code treatments, different layouts — plus a light/dark axis where duels alternated modes and the harness tracked per-mode records. Thirty votes later the data showed something no one would have articulated: one theme was undefeated in both modes (it shipped), another was a dark-mode specialist (4-0 dark, lost in light), a third the exact mirror. Total: 130 votes across three rounds, each round's harness and roster rebuilt from what the previous round's votes revealed.

The rules that make votes cheap and honest

Options must differ structurally. If two options could be confused at arm's length, you're spending votes on noise. Hold the subject constant — vary only the approach, or the comparison teaches nothing (the same rule as the 9x image-mockup pattern in the Variations section). Never force a choice: both-bad is a real signal (it says the whole region of design space is wrong), and quitting early must be free — votes stop being honest the moment they feel like a chore. Record every axis you care about (mode, device width, section) on the vote itself; splits you didn't record are insights you lost. Elo over knockout: pairing least-played, closest-rated options keeps every duel informative, and ratings survive an unfinished session. And notes beat scores — a one-line "C's header wins but B's spacing is calmer" is worth ten silent votes to the agent reading the file.

Beyond design: any decision an agent can render

Nothing in the kit knows about CSS. An option is any file an iframe can show — so the same loop ranks error-message copy, README structures, CLI output formats, API response shapes, function names, landing-page headlines, chart styles. The agent's job is always the same three steps: render genuinely distinct options, stay out of the way while you vote, then read votes.jsonl and turn your picks into rules it applies — and into a DECISIONS.md the next session inherits. That last step is the compounding one: taste captured once stops being re-litigated forever.

Appendix

Limits & caveats

  • Small vote counts are directional, not statistical. 30 votes over 12 options ≈ 5 games each; Elo with K=32 at that depth is a rough sort. Treat standings as "top cluster / bottom cluster", not a precise order.
  • The pairing heuristic (least-played, closest-rated, no immediate repeats) is not true adaptive sampling; informative pairs can be missed. In the case study, the #1 and #2 finishers never met head-to-head — read the pairings before over-trusting a margin.
  • One voter = a personal taste model, not user research. It optimizes for your preference, which is the point — but don't present the output as market evidence.
  • Iframe rendering means options should be same-origin and self-contained; cross-origin options lose scroll-sync and the mode toggle (both fail silently by design).
  • The light/dark axis requires options to respond to <html data-mode="...">; options styled with only prefers-color-scheme won't flip inside the harness.
  • "Both bad" applies a flat −12 to both options — a pragmatic penalty, not principled Elo.

Source references