Tool roles, alternatives, selection tradeoffs, and compatibility caveats.
Preference capture / personal RLHF (this-or-that)
John uses: the this-or-that kit (section 10; kit published at gentle-lasso-9d5n.here.now/kit/) — compare.html + a 30-line server.py + options.json: Elo pairing, arrow-key duels, both-bad/skip, per-vote notes, light/dark per-mode tracking; every vote appends to votes.jsonl, which agents read back as explicit taste rules. This repo's own theme was chosen by it (130 votes, 3 rounds).
| Alternative | What it is | How it differs | License |
|---|---|---|---|
| FastChat (Chatbot Arena) | The platform behind Chatbot Arena: side-by-side battles + Elo ratings for LLM outputs | The industrial version of the same idea (pairwise vote → Elo); heavyweight, model-comparison focused; check current repository activity | Apache-2.0 |
| Argilla | Collaboration tool for building high-quality AI datasets, including preference/ranking records | Team-scale dataset curation with a server; overkill for a solo taste record but the real thing for RLHF data | Apache-2.0 |
| Label Studio | Multi-type data labeling tool with pairwise-comparison templates | General annotation platform; pairwise comparison is one template among many | Apache-2.0 |
| potato | Portable academic annotation tool (Python), supports pairwise/best-worst schemes | Lightweight and self-hostable like the kit, config-driven rather than iframe-anything | GPL-3.0 |
| promptfoo | Prompt/agent eval harness with side-by-side output comparison views | Automated + human-graded evals over prompt variants — the 'evals' complement to taste voting | MIT |
Pick this if…
- the kit itself — it's ~3 small files and free; there's little reason not to start with it.
- promptfoo — your comparisons are prompt/agent outputs and you want automation around the human votes.
- Argilla / Label Studio — you're collecting preferences from a team, not just yourself.
- FastChat — you're literally comparing models and want battle-tested Elo infrastructure.
- potato — you want a tiny self-hosted study-style setup with config-defined schemes.