Most design benchmarks show people two outputs and ask them to vote. We ran ours inside the 21st catalog, where developers choose components for their own projects.
Why copies on 21st are a useful signal
21st is a marketplace of React components. You open a component's preview and copy it, most often as a prompt you paste into a coding agent like Claude Code or Cursor. Copying has a cost. Signed-out visitors have to sign in first, free accounts get two copies a day, and unlimited copying needs a paid membership, which more than 10,000 people have.
So a copy means someone chose this component over the others on the page and spent a limited free copy or a paid subscription on it. In effect they vote with money, and they don't know they're voting. The hero page gets about 48k visitors a month, which is enough traffic to compare models this way.
The models and the prompt
We used 12 models, each at its highest reasoning setting that still finished the job: GPT-6.1 Sol, GPT-6 Astra, Claude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5, Gemini 3.8 Flash, Grok 4.7, GLM 5.3, Qwen 3.8 Max, Kimi K3, MiMo v2.6 Pro and Muse Spark 1.3.
Every model got the same system prompt. It asks for two files in the format 21st publishes (a component and a demo), React 19 with Tailwind v4, a short list of allowed libraries (Framer Motion, GSAP, Three.js, Lucide, a few shadcn/ui primitives), no external images, fonts or other URLs, so every visual is drawn in code, and a layout that works from 375 px wide with no console errors.
The four briefs and what the models made
Ledger
A SaaS product hero (dark):
Build a hero for "Ledger", invoicing software for freelancers. Headline "Get paid without the chasing", subheadline "Send invoices in seconds, track every payment, and let reminders do the awkward part.", call to action "Start free". Everything else (layout, visuals, colors, typography, motion) is up to you.
Ledger, 12 models
Aurelia
A scroll-expanding media hero (dark):
Build a hero for "Aurelia", a travel film studio. A framed media panel sits in the center of the first screen with the title "Stories from the edge of the map" split around it. As the user scrolls, the panel grows until it fills the viewport, the two halves of the title slide apart and fade, and the page then continues to a short paragraph: "Twelve films. Six continents. One crew that never stops moving." The media panel is a slow, abstract animated landscape drawn in code. Mood: cinematic, quiet.
Aurelia, 12 models
Orbit
An interactive 3D hero (follows the site theme):
Build a hero for "Orbit", a satellite-data API. Left: eyebrow "Now in public beta", headline "The planet, as an API", subheadline "Query imagery, weather and ground truth for any coordinate in a single call.", buttons "Get an API key" and "Read the docs". Right: an interactive 3D globe or object that slowly rotates and tilts toward the cursor. On mobile the object moves above the text. Mood: precise, technical.
Orbit, 12 models
Nocturne
An atmospheric dark hero:
Build a hero for "Nocturne", a sleep-sound app. A dark, atmospheric scene with a slow light effect (beams, glow, aurora or stars, your choice) behind very large type: "Fall asleep to something beautiful". Below: "Generative soundscapes that never repeat, tuned to how you breathe." and buttons "Try it free" and "Listen to a sample". Elements reveal in sequence on load. Mood: dreamy, slow, premium.
Nocturne, 12 models
Getting a working component out of each model
We rendered every output in a headless browser at 375 and 1440 px, in light and dark mode. A model got another attempt only when its first one didn't work: no code, a crash, or a response cut off mid-stream. The first working attempt is the one that went live. Each model had to return its two files as markdown code blocks, and when a model broke that format the code couldn't be extracted automatically, so we fixed it by hand.
| Model | Cost for 4 heroes | Typical time per hero | What went wrong |
|---|---|---|---|
| Claude Fable 5.1 | $15.96 | 17 min | Orbit took four attempts. Two responses were cut off, and on the third it spent all 128k output tokens reasoning and wrote no code ($6.41) |
| GPT-6 Astra | $8.09 | 17 min | One Ledger response cut off mid-stream, retried |
| Claude Opus 5.5 | $5.71 | 11 min | At the maximum reasoning level it used all 128k tokens thinking and returned no code, so it ran one level lower. One Orbit response was cut off and retried |
| Grok 4.7 | $2.41 | 22 min | Slowest: up to 30 minutes and 137k reasoning tokens for one hero |
| Claude Sonnet 5.5 | $1.71 | 6 min | Same as Opus at the maximum level, so it also ran one level lower |
| GPT-6.1 Sol | $1.50 | 14 min | One response cut off mid-stream, retried |
| Kimi K3 | $1.39 | 9 min | The Orbit globe renders as a flat grey disc |
| Qwen 3.8 Max | $1.38 | 7 min | One Ledger response cut off mid-stream, retried. The first Orbit attempt crashed on load (useRef used without importing it) |
| GLM 5.3 | $1.21 | 9 min | On Orbit it never closed its code block and put the file name inside it, so the code was fixed by hand. The Aurelia panel is nearly empty after scrolling |
| Muse Spark 1.3 | $0.41 | 2 min | On Ledger it never closed its code block, so the code was fixed by hand |
| Gemini 3.8 Flash | $0.22 | 1.5 min | Broke the code-block format on Ledger and put the file name inside the code block on Orbit, both fixed by hand. On Ledger it also imported shadcn components that 21st's publisher couldn't resolve |
| MiMo v2.6 Pro | $0.10 | 10 min | The first Orbit attempt crashed on load. The Aurelia panel goes blank after scrolling |
All 48 components ended up working. The cost for one hero ranged from $0.02 (MiMo) to $4.53 (Fable), and the whole run came to about $50 including failed attempts. More reasoning made the code longer, usually 600 to 1,100 lines, but it didn't make the designs more different from each other.
How the test works
The 48 sections are published from one neutral studio account with matching covers, and the cards don't show the model. Signed-in visitors to the hero page see two of them in the feed, at positions 6 and 12. Both come from the same brief and from two different models, so every page view is a blind comparison between two models. A random 10% of visitors get the normal feed as a control.
The hero page as a test-group member sees it, with the two AI sections outlined in slots 6 and 12
How the sections did
Cards higher in the feed get opened more, so we compare each AI section with the human card that the control group saw in the same slot. Both sides count only the default view of the hero page, and each member once.
| Position | What was there | Who saw it | Saw it | Opened it | Open rate |
|---|---|---|---|---|---|
| 6 | AI section | test group, 90% of visitors | 6,545 | 242 | 3.7% |
| 6 | Human-made card | control group, 10% of visitors | 708 | 26 | 3.7% |
| 12 | AI section | test group, 90% of visitors | 5,331 | 163 | 3.1% |
| 12 | Human-made card | control group, 10% of visitors | 363 | 8 | 2.2% |
The AI sections hold up against human-made cards: the gap is within noise in both slots. They didn't put people off the page either. The share of visitors who open anything at all is 54% with the inserts and 54% in the control.
The brief made a bigger difference than the slot did.
Aurelia's lead is statistically real (p ≈ 0.005). It is also the brief where the models' designs differed the most, while on the other three most models converged on the same layout.
Which models won
The models do differ overall (p ≈ 0.02 across all twelve), but no single leader is ahead of the rest beyond the margin of error.
We rank the models two ways. Pick rate is the share of members who opened a model's section after seeing it. Take Claude Fable 5.1: 1,332 members saw at least one of its four sections, and 40 of them opened the one they saw, so its pick rate is 40 / 1,332 = 3.0%. Each member counts once per model. Some briefs get opened more than others and they didn't land evenly across models, so the rates below are adjusted for each model's mix of briefs.
Head to head uses only page views where a member saw both sections and opened exactly one: the opened one wins. A Bradley–Terry model turns those wins into ratings on the Elo scale, where 1500 is the average model, and takes out the advantage of sitting in slot 6.
The two rankings agree at both ends: GPT-6.1 Sol is first and Kimi K3 last on both, and Claude Fable 5.1 is third on both. In the middle they disagree most about Gemini 3.8 Flash, which is seventh by pick rate but second head to head. Price doesn't buy a better result here: GPT-6.1 Sol cost $1.50 for its four heroes and Claude Fable 5.1 cost $15.96, and Gemini 3.8 Flash, at $0.22, is within the leader's margin of error head to head. There were 305 duels, at least 40 for every model. On both rankings the leader is not clear of the next few models, but MiMo v2.6 Pro and Kimi K3 trail it on both. Copies are too rare to compare models: 26 in total.
All the numbers in one table:
| Model | Pick rate (95% interval) | Members who saw it | Rating (95% interval) | Duels | Won |
|---|---|---|---|---|---|
| GPT-6.1 Sol | 3.6% (2.7–4.8) | 1,324 | 1617 (1531–1703) | 61 | 66% |
| GPT-6 Astra | 3.5% (2.6–4.6) | 1,383 | 1521 (1439–1602) | 62 | 53% |
| Claude Fable 5.1 | 3.0% (2.1–4.1) | 1,332 | 1549 (1453–1645) | 46 | 54% |
| Claude Sonnet 5.5 | 3.0% (2.1–4.0) | 1,366 | 1502 (1411–1593) | 51 | 51% |
| Grok 4.7 | 2.8% (2.0–3.8) | 1,365 | 1492 (1403–1580) | 54 | 50% |
| Muse Spark 1.3 | 2.7% (1.9–3.8) | 1,333 | 1484 (1395–1573) | 53 | 47% |
| Gemini 3.8 Flash | 2.7% (1.9–3.7) | 1,355 | 1558 (1467–1649) | 52 | 58% |
| Claude Opus 5.5 | 2.3% (1.6–3.3) | 1,340 | 1520 (1425–1615) | 47 | 51% |
| Qwen 3.8 Max | 2.3% (1.5–3.3) | 1,299 | 1481 (1389–1573) | 50 | 44% |
| GLM 5.3 | 2.1% (1.4–3.0) | 1,372 | 1473 (1381–1565) | 50 | 44% |
| MiMo v2.6 Pro | 1.8% (1.2–2.7) | 1,346 | 1425 (1320–1529) | 40 | 40% |
| Kimi K3 | 1.6% (1.0–2.4) | 1,324 | 1380 (1278–1482) | 44 | 34% |
The spread between sections is wide. Grok 4.7's Aurelia, the most opened section overall, was opened by 19 of the 382 members who saw it (5.0%). Kimi K3's Nocturne, the least opened, was opened by 1 of 355 (0.3%), and MiMo v2.6 Pro's and Qwen 3.8 Max's Ledger also have a single open each (1 of 338 and 1 of 309). Even within one brief the gap is large: GLM 5.3's Aurelia was opened by 4 of 348 (1.1%).
Grok 4.7's Aurelia, the most opened section, next to Kimi K3's Nocturne, the least opened one