Habit, a trial · installs only where you enable it

benchmark-checklist

about 1,700 tokens

benchmark-checklist as it installs, each line coloured by where its words came from.

  • 2 files
  • 0 edits
  • 24 sentences left out
  • 1 source
  • last changed 2026-10-19
  • trial

The text

SKILL.md

about 1,600 tokens
  • pstack
  • greenline

---

name: "benchmark-checklist"

description: "Vet a measured number before you report or act on it: name the limiter, check the sides were tuned, do the arithmetic against physical limits, count the errors, run each side at least five times alternating them, weigh the result end to end and confirm the work happened, then report the verdict with the median and the range. Use when a performance or eval number is about to be reported, or to decide a choice."

---

pstackbenchmark-checklist/SKILL.md

# Benchmark checklist

greenlineOpens the trial habit: vet a performance number before reporting or deciding

benchmark-checklist is a trial habit, installed only where the workspace's `trials` names it, for the moment a performance or eval number is about to be reported, or to decide a choice: a library or configuration choice at frame, a performance regression in diagnose, a speedup or a regression at verify. It writes no record of its own: the number goes into the section of the step that measured it, with its run count, its range and its limiter, and the runs it reports are checks, run as the verify step runs its own. For a measured number its alternating runs replace the verify step's single baseline then treatment; every other claim keeps the verify step's method. Proving that an output is real is the verify step's, and finding the cause of a regression is diagnose's.

greenlinePoints at the skill's references page for paths and commands

Greenline's paths and commands for this skill stand on its references page, [references/greenline.md](references/greenline.md): read it when the work lands in the workspace.

pstackbenchmark-checklist/SKILL.md

Use this when you produce a performance number: a PR's before and after, a regression claim, a hillclimb harness, or a library or config choice.

pstackprinciple-explain-the-number/SKILL.md

**Why:** A run that went wrong still prints a plausible number. Requests that failed, a cache that skipped the work, code that never ran, a side left on default settings, and run-to-run noise all produce results that look fine. If you cannot say why the number is not twice as good, you do not know what you measured.

pstackbenchmark-checklist/SKILL.md

Answer each question below with evidence from a run, not from a guess about the code.

For a quick ballpark the user asked for, one run is enough. Still check questions 4 and 7, and say that it is one run. Skip the rest unless that run looks wrong. A choice between options is never a ballpark.

## Before you run anything

- Write down the claim you expect to make, in the words you would ship ("export is 30% faster at p50 on the 60k-row dataset"). The questions test that sentence.

- Read the measurement script. Note what it times, what it counts, and what it ignores.

- Check the load average with `uptime` and the core count with `nproc`. If the machine is busy, find out what is running. If you cannot stop it, interleave the sides so both see the same noise, and say so in the report.

## The questions

1. **Why not double?** Name the limiter. Profile in a run you do not report, because profilers and tracers slow the work down. Use CPU per process (`top`, `pidstat`), a profiler for the runtime (`node --cpu-prof`, `py-spy`, `perf`), I/O wait, and syscall counts (`strace -c` on Linux). Then map the hot spot to source. Watch the load generator too. If it saturates first, you measured the load generator. If a change did not move the number, the limiter explains why, so find it before you call the change useless.

2. **Was it tuned?** Run every side the way production runs it: release builds, production flags and env, batching and transaction settings, connection pools, caches as warm or cold as production sees them, and the same versions and data. If one side runs on defaults, you compared configurations, not implementations. A limiter that is a setting, such as a commit per row, a debug build, or a missing index, means that side is untuned. Tune it and measure again before you pick a winner. If you cannot tune it, do not pick a winner from that run. Narrowing the claim to the code as it ships today does not fix this when the user is choosing what to adopt, because they adopt the option, not today's settings.

3. **Did it break limits?** Do the arithmetic. Compare bytes per second with disk and network bandwidth, and operations per second times the cost per operation with the cores you have. Compare the time saved with the time the changed piece took. Removing a piece that takes 10% of the run can make the run at most about 11% faster. A result past a limit means the run measured something other than the work, such as a cache, a no-op, or a bug.

4. **Did it error?** Count failures and non-success responses, and check that the outputs are correct, not just present. Errors behave differently from successes. Rejections are often fast, and timeouts and retries are slow. If the script does not count errors, add the count.

5. **Does it reproduce?** Run each side at least 5 times, and alternate the sides (A, B, A, B, and so on) so that warmup, lazy initialization, caches, and drift do not favor one side. Report the median and the range. Treat a gap smaller than the run-to-run variation as no measurable difference. When the call is close, use a rank-sum test or the harness's own statistics.

6. **Does it matter?** Next to any micro result, measure the end-to-end path a user waits on, with realistic data sizes and concurrency. Report the micro result as a share of the whole. A helper that takes 1% of a request can make the request at most 1% faster, however fast the helper gets.

7. **Did it even happen?** Confirm the work ran inside the timed region. The request reached the server, the rows were written, the bytes were read, and the code used the result. Lazy code (generators nobody iterates, promises nobody awaits, results the JIT can discard) and timeouts all produce numbers for work that never happened.

pstackprinciple-explain-the-number/SKILL.md

For an eval result, ask the same of the trials: did every run do the task, does the gap hold across trials and models, and does the scenario matter.

pstackbenchmark-checklist/SKILL.md

## Report

- Lead with the verdict: faster, slower, no measurable difference, or inconclusive.

- Give the number with its unit, the run count, the range, and the limiter. For example, "p50 41 ms → 33 ms, median of 7 runs per side, range 32 to 35 ms after, bound by JSON parsing on one core."

- Call the verdict inconclusive when you claim a difference but cannot name the limiter, when a side ran untuned, or when you could not check questions 4 and 7. Name the gap.

greenlineEvidence goes in the change's verify folder, cited from the step's section

- Keep a PR body to one primary number. Put the runs, the range and the limiter evidence in the claim's evidence in the change's folder, as the verify step promotes its evidence, and cite them from the step's section.

greenlineMaps its four verdicts onto the verify step's three

In the verify step the verdict is one of its three: faster or slower as the claim predicts is `VERIFIED`, the other direction, or no measurable difference where the claim named one, is `NOT VERIFIED`, and inconclusive is `INCONCLUSIVE`.

pstackprinciple-explain-the-number/SKILL.md

You skipped this when the evidence behind a number has no run count, no spread, or no named limiter, or when the time saved is larger than the time the changed piece took.

SKILL.md as installed, byte for byte SHA-256 32a7ae588e0687a068c9811c67558163e3f1231d50f257b2a625d0fb5e0f6de3

references/greenline.md

about 110 tokens
  • greenline
greenlineOpens the skill's mechanics page, read with SKILL.md

# Benchmark checklist: the practice's mechanics

This page holds where benchmark-checklist's work lands in the workspace and the commands it runs, which `SKILL.md` names by their nouns. Read it with `SKILL.md` when the work lands in the workspace.

greenlineWhere a benchmark's runs and evidence land

## Where its work lands

A benchmark's runs, its range and its limiter evidence go in the claim's evidence, `.greenline/ledger/<id>/verify/<claim-slug>/`, as the verify step promotes its own.

What greenline wrote

The full reason for every passage greenline wrote, in the order of the text.

  1. SKILL.md:8 scope

    One opening paragraph in the habit's own voice: the situations its block row keys it by, that it is a trial installed only where trials names it, that it writes no record and where the number and its runs go, the ruled contradiction with the verify step, and the step and the start beside it.

  2. SKILL.md:10 pointer

    The front points at the skill's references page, where greenline's paths and commands for it stand since J-93 S2, so the front holds the method alone.

  3. SKILL.md:43 relocate

    The evidence lives in the change's folder the verify step promotes into, cited from the record, in place of a linked artifact, a notes file or a playbook the installation does not carry. Since J-93 S2 the front names greenline's homes and commands by their nouns, their paths and commands on the skill's references page.

    Credits: pstack

  4. SKILL.md:45 scope

    The habit's four verdicts map onto the verify step's three, so a number reported in the verify section reads in that step's words.

  5. references/greenline.md:1 scope

    The skill's references page opens: greenline's mechanics for the skill, the paths and commands its front names by their nouns, held apart from the upstream prose since J-93 S2.

  6. references/greenline.md:5 relocate

    benchmark-checklist's mechanics: where a benchmark's runs and evidence land, held apart since J-93 S2.

What it left out

pstack

pstack, benchmark-checklist/SKILL.md: 9 of 70 sentences left out
  1. [Explain the Number](../principle-explain-the-number/SKILL.md) says why. left out: The pointer to Explain the Number gives way to its why, grafted in its place. replaced by line 14 of SKILL.md
  2. 59 sentences taken
  3. Keep a PR body to one primary number, per the **Opening a PR** playbook.
  4. Put the runs, the range, and the limiter evidence in a linked artifact or a notes file. 2 sentences left out: The PR body's one number stands; the playbook it names is not carried, and the evidence goes to the change's folder. replaced by line 43 of SKILL.md
  5. ## How this fits the other perf material
  6. The **Perf issue** playbook finds and fixes slowness, and the performance mantras in its step 2 generate the fixes.
  7. This skill vets its baseline before the playbook plans from it, and every number after that.
  8. The **Hillclimb** playbook loops on one metric.
  9. This skill vets its harness before the harness is frozen.
  10. The frozen harness then prints error and work counts, so each keep-or-revert checks questions 4 and 7 for free. 6 sentences left out: The perf-issue and hill-climb playbooks are not carried; the opening paragraph names where the habit runs instead, beside diagnose and the verify step. replaced by line 8 of SKILL.md
pstack, principle-explain-the-number/SKILL.md: 15 of 20 sentences left out
  1. # Explain the Number
  2. A measured number is a claim about a system.
  3. Before you trust it, report it, or act on it, find what limits it and rule out that it measured something else. 3 sentences left out: The title and the claim that a number must be explained before it is trusted are the spine's own opening and questions.
  4. 3 sentences taken
  5. **Pattern:**
  6. **Ask "why not double?"**
  7. Name the resource or code path that bounds the result, such as a core, a lock, the disk, the network, or the load generator itself.
  8. Get it from a profile or from system counters taken during a run, then map it to source.
  9. A guess from reading the code is not a limiter.
  10. **List what else the number could be measuring, and rule out each one with evidence.**
  11. The usual suspects are errors, skipped or cached work, an untuned side, noise, and a piece too small to matter end to end.
  12. **Keep the evidence with the number.**
  13. Put the run count, the spread, and the limiter in the notes or a linked artifact, so a reader can check the claim. 9 sentences left out: The pattern duplicates the spine's questions: why not double is question 1, the other suspects questions 2 to 7, and the evidence kept with the number the report. replaced by line 16 of SKILL.md
  14. For a performance number, run the full procedure with the [benchmark-checklist](../benchmark-checklist/SKILL.md) skill. left out: The pointer to benchmark-checklist points at this skill itself.
  15. 2 sentences taken
  16. Distinct from [Prove It Works](../principle-prove-it-works/SKILL.md), which checks that an output is real.
  17. This checks that a measured number means what you say it means. 2 sentences left out: The pointer to Prove It Works becomes the opening paragraph's neighbour, the verify step. replaced by line 8 of SKILL.md

The ledger entry

greenline's ledger logs how each skill was made and changed, with a reason for each entry.

2026-10-19 2026-10-19-benchmark-checklist-front-and-references

benchmark-checklist's paths and commands move to a references page

The front names greenline's homes and commands by their nouns and points at a references page, which holds their paths and commands (J-93 S2).

Authorised by
2026-10-18 The operator's words of 2026-10-18 on the redesign (N-56, items 2 and 7), "we need the upstream prose upfront and references hold how to work meaning the artefacts etc" and, on the ingest procedures, "we wanna have there prose .. what we don't wanna have is there artefact system and fit them to use ours"; and J-93 S2's finalise words of 2026-10-19, "Fast route, merged slices, parallel jobs (Recommended)": a skill's front is the upstream prose its recipe cuts, and its mechanics, where its work lands, its section's form, its depths at each mode and its commands, stand on its references page
Pages it touched
SKILL.md, references/greenline.md
Rulings, R1 to R4
  1. the operator's rulings of 2026-10-08 on the intake R3 ("everything is good", and "yes go ahead" on the portable forms of R21), the verdicts standing and every contradiction ruled as recommended: contradiction 12 as recommended, benchmark-checklist's alternating sides for a measured number and verify's method for every other claim; its PR and hill-climb playbooks dropped
  2. under R22 of 2026-10-08, the re-cut brief accepted as recommended ("yes amend, with one change"): the names of the roster table and the trial names stand, and show-me stays implicit
  3. J-59, on the operator's words of 2026-10-09 ('okay then implement all of this'): a gist of at most twelve words on every account whose reason is stated; no text, measure or method changes
  4. The operator's words of 2026-10-18 on the redesign (N-56, items 2 and 7), "we need the upstream prose upfront and references hold how to work meaning the artefacts etc" and, on the ingest procedures, "we wanna have there prose .. what we don't wanna have is there artefact system and fit them to use ours"; and J-93 S2's finalise words of 2026-10-19, "Fast route, merged slices, parallel jobs (Recommended)": a skill's front is the upstream prose its recipe cuts, and its mechanics, where its work lands, its section's form, its depths at each mode and its commands, stand on its references page
2 earlier entries

  1. 2026-10-09 2026-10-09-benchmark-checklist-gists

    Every account gets a short gist for the website

  2. 2026-10-08 2026-10-08-benchmark-checklist-recipe

    Enters as a trial habit for measured numbers

    Pages it touched: SKILL.md