Skip to content
DurkBench

Method history

Changelog

Every change to how we test, in the open and dated. A benchmark that edits its own rules quietly is not a benchmark.

The rules in force today

What holds right now, so you do not have to read the history below to find out.

How to read the tags

Test kit vN
A breaking change to how we measure. Older rows leave the boards that moment, but they stay in Logs forever.
Cadence
A safe change — how often we run, or which models are on the roster. Old and new samples still compare.
Quality vN
The version of the graded question set the lab ran before it retired. Pass rates compare only within one version.
No rescoring
A sample keeps the grade it earned under the rules of its day. We never rewrite history.

What changed, and when

Newest first. The tag says whether older numbers still compare, and the line under each card says what the change did to the live boards.

  1. August 20, 2026Cadence

    Slower pulse: every model, every hour

    • The lab now runs one round per hour, at the top of the hour. Before, it ran every 15 minutes.
    • Every pinned model still runs in every round, at the same effort, with the same prompts.
    • Fewer rounds means fewer samples per day. The medians need a little longer to settle after a model changes.

    Test kit unchanged, so samples from before and after this date still compare.

  2. August 20, 2026Test kit v6

    A warning is not a refusal

    • Claude's app sends a typed note when a plan's usage window is getting full. We were counting that note as a rate limit, even though every one of those runs was served and answered correctly.
    • Now only a real refusal counts as a rate limit. A served-but-warned round gets its own label, rate limit warning, and its own count on the boards.
    • Warned rounds still stay out of the speed medians. In our logs they ran about 15% slower than clean rounds, so mixing them in would drag the numbers.
    • A warning no longer stops the rest of that app's models for the round. Only a real refusal does.

    Breaking change. Boards use v6 rows only. Everything older stays in Logs and never mixes in.

  3. August 19, 2026Cadence

    The graded-answer benchmark is retired

    • We no longer run the 24-question graded suite, and its board is gone from the site. The address /quality now sends you to How we test.
    • Speed is the benchmark this site publishes. Speed rounds are still graded: a round only counts when the answer is right, so a fast wrong answer still scores nothing.
    • Old graded results are not deleted. They stay in our saved logs with the test kit version they ran under. We just do not publish a board for them anymore.

    No speed number changed. Test kit unchanged — every number from before and after this date still compares.

  4. August 19, 2026Cadence

    Boards now lead with four featured models

    • Every board now leads with Fable 5, Opus 5, GPT-5.6 Sol, and Grok 4.6.
    • The other six pins — Haiku, Sonnet, Luna, Terra, GPT-5.5, and Grok 4.5 — moved to the Other models page.
    • Nothing changed in how we measure. Every pin still runs in every round.

    Presentation only. Test kit unchanged — old and new numbers still compare.

  5. August 19, 2026Test kit v5

    Top effort for Grok, and a retired model

    • Grok 4.6 now runs at xhigh, the top of its effort scale. Claude and Codex already ran at high, the top of theirs. Grok 4.5 stays at high because that is its ceiling.
    • Fixed our reader for Grok's new streaming output. Grok's answers were arriving fine, but our reader saved them as empty, so every Grok speed run failed grading. Those failed slots stay failed — we never rescore.
    • GPT-5.2 is retired from the roster. OpenAI's backend now refuses it on ChatGPT subscription accounts, so it cannot be measured the way we measure everything else.

    Breaking change. Boards use v5 rows only. Everything older stays in Logs and never mixes in.

  6. August 19, 2026Cadence

    Faster pulse: every model, every 15 minutes

    • The lab now checks every pinned model every 15 minutes. Before, cheap models ran each round and big models ran every 2 to 6 hours.
    • Effort stays high on every pin.
    • New: the Share your take button. Visitors can tell us how a tool feels. Those votes get their own board and never mix into lab numbers.

    Test kit unchanged, so samples from before and after this date still compare.

  7. August 2026Test kit v4

    Clean rooms and graded answers

    • Each run starts in a fresh empty folder with no extra settings, skills, or tools.
    • Answers are graded. A run only counts when the answer matches what we asked for.
    • Rate limits are read from each app’s own typed events, not guessed from the text on screen.

    Breaking change. Boards use v4 rows only. Everything older stays in Logs and never mixes in.

  8. August 2026Quality v2

    A graded suite of its own

    • 24 short prompt tests with one clear right answer each, run four times a day.
    • Quality gets its own board and its own window.

    Quality is never blended into speed. A fast wrong answer stays wrong.

Read next