Explainer · companion to the map
Verification Gaps
What Crank’s checks don’t cover yet
The Agent Engineering Map shows how agents prove their work. This page is the other half: eleven checks Crank doesn’t have yet, in three groups: verification, quality and craft. Ten don’t exist at all, and one runs for motion graphics only. Each gap is sorted by kind: one right answer means a program can check it, and judgement means it needs scored examples and a second opinion. Every term is explained the first time it appears.
- Group V · verification Does it work? 1 CI on every PR 2 Preview links 3 Screenshot compare 4 Live checks 5 Env preflight
- Group Q · quality Is it right? 6 Incoming data 7 Figure tracing 8 Judgement scores 9 Repeatable builds
- Group C · craft Does it feel like Crank? 10 Style rules 11 Second-model critic
How to read it: each group starts with two diagrams and a short explanation, then one card per gap: what’s missing, why it matters, how it could be closed and where it would live. Every card is a gap, so nothing here claims the fix exists.
Tool one right answer, so a program can check it. Judgement needs scored examples and a critic. Crank practice marks decisions Peter and Cal have locked for Crank.
Three ideas first
Check
A test that runs on every piece of work and says pass or fail, so nobody has to remember to look. A check that passes when it shouldn’t is the worst kind, because everyone trusts it.
Gap
A kind of mistake that can reach Peter, or a client, with nothing catching it first. Each gap on this page is a check Crank doesn’t have yet.
Tool or judgement?
If two careful people would always agree on the answer, a program can check it: that’s a tool. If they could fairly disagree, like whether a sentence reads well, it takes judgement: scored examples and a second opinion.
In the map’s kitchen picture, these are tasting steps the kitchen doesn’t have yet. The map’s Pillar C explains how an agent proves its own work. This page lists what nothing checks today. The list of eleven comes from Cal, Forge and miniForge in Crank Tools.
Group V · gaps 1 to 5
Does it work? · verification
Checks that run on every change, without anyone remembering to run them.
CI (continuous integration) runs a project’s tests every time a PR opens or changes. If a test fails, the PR is marked as failing. A preview link is a working copy of a draft that you can click before it merges. A screenshot comparison captures a page, a frame or a slide and compares it with the version Peter approved. A broken layout then fails, even when every test passes. Post-release checks open the live page or service after each release and read what comes back. An env preflight checks that the keys a job needs are there, and the right shape, before it starts. Env is short for environment: the settings a program starts with.
Crank practice A passing check never merges anything. Agents open PRs as drafts and a person does every merge, so checks change how closely Peter looks, not who merges. For UI products, the official verify path stays pstack’s /create-verification-skill and /maintain-verification-skill. Gaps 2 and 3 would feed that path, not replace it.
Gap 1: CI checks on every PR
- What’s missing
- Nothing runs a repo’s tests when a PR opens or changes. A PR can say “done” with failing tests, and nobody finds out until something breaks.
- Why it matters
- Agents open a lot of PRs. Without CI, every “it works” rests on the agent’s word or on Peter’s time.
- Closing it
- Run the build and the tests on every PR, and mark the PR as failing when they fail, so it can’t merge until it’s fixed. A pass doesn’t merge anything: a person still does.
- Would live in
- Each repo’s own setup, next to its tests.
Seen 7 Oct 2026: none of this repo’s five PRs has run a test check. The only check reported is Cursor Bugbot, a code-review bot, on PR #3.
Gap 2: A preview link for every PR
- What’s missing
- A draft PR has no working link. To see a change before it merges, someone has to build it themselves.
- Why it matters
- Peter reviews by looking. A link on every draft lets him, or a checking bot, try the real thing instead of reading about it.
- Closing it
- Build a preview for every PR and post its link on the PR. The check fails if the link doesn’t load, or loads the wrong page.
- Would live in
- Each repo’s own setup. For this site, a Cloudflare Pages preview for each branch.
Seen 7 Oct 2026: the preview address for PR #3’s branch answers “Deployment Not Found”.
Gap 3: Comparing screenshots to an approved version
- What’s missing
- Nothing compares how a film frame, a page or a slide looks now with how it looked when Peter approved it.
- Why it matters
- A change can pass every test and still look wrong: a shifted logo, a cut-off caption, the wrong colour.
- Closing it
- Save an approved screenshot, the baseline, for each frame, page or slide. Capture each new version the same way and compare them: a difference above a small limit fails and shows both side by side. When a change is meant to happen, a person approves the new screenshot as the new baseline.
- Would live in
crank-tools, as one shared check every agent can call (not built, no name yet).
Gap 4: Checks after a release goes live
- What’s missing
- After a release, nothing opens the live site or service to confirm it works. If a deploy fails or serves the wrong thing, nobody is told.
- Why it matters
- Failures stay silent. The worst ones look fine to a simple check, like the missing page in V2 that answers “OK”.
- Closing it
- After each release, open a short list of key pages and services and read what comes back: the right title or marker, not just status 200. Raise an alert if anything is wrong.
- Would live in
crank-toolsfor the check (not built, no name yet), with each repo listing its own key pages.
Seen 7 Oct 2026: crank-experiments.pages.dev answers status 200 with the hub for a made-up address, and for the map page, which isn’t deployed yet. It also serves techniques 41 to 44, which exist only on PR #1’s draft branch, not on main. Nothing flagged either.
Gap 5: Env preflight
- What’s missing
- Jobs start without checking that the keys and settings they need are there. A missing or mistyped key fails halfway through, often with a confusing error.
- Why it matters
- A job that fails late wastes the run and can leave half-finished output behind.
- Closing it
- Before a job starts, check that each required key is present and the right shape (its length or pattern), without reading or printing the value. If the setup is wrong, the job doesn’t start: in the gap list’s words,
wrong way fails a check
. - Would live in
crank-tools, as one preflight every job calls before it starts (not built, no name yet).
Raised by: miniForge, in Crank Tools.
Teachers: @poteto, through pstack (tests alone are not sufficient verification
); Cloudflare’s Pages docs (how missing pages and previews behave). The gap list itself is Crank’s.
Go deeper: verification gaps
- CI is the floor, not the proof. pstack’s planning playbook tells agents that
tests alone are not sufficient verification
. A green CI run says the tests passed. Showing that the feature works is the map’s Pillar C. - Previews for this site: Cloudflare Pages’ Git integration connects to GitHub or GitLab, and Crank’s repos live on Origin, so Pages can’t build a preview for each PR by itself. A CI job could upload each branch instead (
wrangler pages deploy dist --branch <branch>), which gives a preview at<branch>.crank-experiments.pages.dev. Cloudflare’s Workers Builds does connect to Origin, and posts preview links and check runs on PRs, but moving the site off Pages would change the stack, so it’s only an option. - Preview links are public by default, Cloudflare’s docs say, and an access policy can limit them to Crank’s Cloudflare account. A preview of client work would need that switched on.
- The hub-for-404 has a simple cause. Pages treats a site with no top-level
404.htmlas a single-page app and answers every unknown address with the home page. Adding a404.htmlwould make missing pages return a real 404. A live check should still read the content, because a page can answer 200 and still be the wrong page. - An env preflight, done safely: check the name is set, the value isn’t empty, and it has the expected length or prefix. Print only the key’s name and pass or fail, never the value or any part of it.
- Blocking a merge needs the forge to support it. Cloudflare’s Origin integration creates check runs that show on Origin PRs. Whether a failing check can block a merge on Origin is still to be confirmed.
Group Q · gaps 6 to 9
Is it right? · quality
Numbers and calls that match the source, today and tomorrow.
Incoming data checks look at data before anything uses it. Is it fresh, from today and not last week? Is it in the expected format? Do two sources that should agree actually agree? Tracing means every number in a deck or brief can be found in the source data, or worked out from it. A typo or a made-up figure fails. Scoring the model’s judgement means keeping examples Peter has already approved. After each change, the model runs on them again, and a score shows how close it gets. Repeatable builds pin every tool to one exact version, so nothing updates overnight. A check that passes today passes tomorrow for the same reason.
Crank practice Deterministic vs Judgement sorts this group. Gaps 6, 7 and 9 have one right answer, so a program checks them and says pass or fail. Gap 8 is judgement: the model makes the call, and scored examples show whether its calls are getting better or worse.
Gap 6: Checks on incoming data
- What’s missing
- Data from outside tools is used as it arrives. Nothing first checks that it’s fresh, in the expected format, and in line with other sources.
- Why it matters
- A stale or broken data pull turns into a confident, wrong sentence in a brief. Later checks can’t catch it: tracing (gap 7) would match the figure to the bad data.
- Closing it
- When data lands, check its date (fresh enough?), its columns and types (right format?) and whether totals that should match across sources do, within a set tolerance. If not, stop and flag it with both numbers side by side. Which source is right is a person’s call.
- Would live in
- Each data job’s own repo, because each job knows its own sources and what “fresh” means for them.
Gap 7: Tracing every figure to its source
- What’s missing
- Numbers in decks and briefs aren’t checked against the data they came from.
- Why it matters
- A model can copy a number wrongly, round it differently or invent a plausible one, and it reads just as confidently as a true one.
- Closing it
- Pull every figure out of the draft and look for it in the source data, or work it out again from source numbers, like a percentage change. Anything not found fails and is listed for a person to check. Q2 shows one.
- Would live in
crank-tools, as one shared check for any deck or brief (not built, no name yet).
Gap 8: Scoring the model’s judgement
- What’s missing
- There’s no fixed set of examples to test the model’s calls against: how it words things, which category it picks, which insights it pulls out.
- Why it matters
- When a prompt, a skill or the model itself changes, nobody can tell whether its judgement got better or worse, only that it’s different.
- Closing it
- Collect examples Peter has already approved and keep them in git. Re-run the model on them after each change and score the results. Categories can be scored right or wrong. Wording and insights are scored against a short checklist, by a critic or a person.
- Would live in
- Next to the skill it tests, with the examples in git.
Gap 9: Repeatable builds
- What’s missing
- Tool and browser versions aren’t pinned. The same check can pass today and fail tomorrow because a tool updated overnight, not because the work changed.
- Why it matters
- If checks flip for reasons outside the work, people stop trusting them, and real failures get waved through.
- Closing it
- Pin exact versions of the runtime, the packages and the browser used for screenshots, and record them with every run. Upgrade on purpose, in a PR of its own.
- Would live in
- Each repo’s own setup: a lockfile for packages, a pinned Node version, a pinned browser.
Seen 7 Oct 2026: this repo accepts any Node version from 18 up, and its preview script downloads whatever version of serve is newest.
Go deeper: quality gaps
- Fresh, right format and in agreement are three separate checks. Fresh compares the newest row’s date with today. Format compares the columns and types with what the job expects. Agreement compares totals that should match, within a set tolerance, and shows both numbers when they don’t.
- Tracing can only prove a figure matches its source. If the source is wrong, the figure matches the wrong number, which is why gap 6 comes first.
- Scoring judgement: once Peter has made a call, like which category an ad belongs in, it can be scored right or wrong. Wording and insights need a rubric (a short scoring checklist), applied by a critic or a person. Keeping the examples in git means a score means the same thing next month.
- Repeatable builds include fonts. A screenshot check depends on the browser version and on which fonts are installed. This site’s own diagram labels measure wider in DejaVu Sans than in Noto Sans.
- In this repo,
package.jsonsets"node": ">=18", andnpm run previewrunsnpx --yes serve, which fetches whatever version ofserveis newest that day.
Group C · gaps 10 and 11
Does it feel like Crank? · craft
House style checked by a program, and a second opinion before Peter.
A style check is a program that reads a draft and flags whatever breaks the house style: a banned word, a sentence too long for plain English, a colour or font that isn’t Crank’s. It has one right answer, so it can run on every draft in seconds. A second-model critic is a different AI model, asked to review the draft against the brief before Peter sees it. It catches what rules can’t: a weak opening, a claim nobody asked for, a frame that feels off. Crank already runs a critic loop for motion graphics. The gap is making independent review the normal step for every draft.
Crank practice A critic’s pass doesn’t approve anything: Peter still sees the draft, and nothing goes to a client without him. When the critic keeps flagging the same thing, fix it at the strongest rung of the map’s correction ladder. A banned-word rule beats another reminder in a skill.
Gap 10: Style rules a program checks
- What’s missing
- Nothing checks plain English, banned words or the Crank look automatically. It’s up to whoever reads the draft.
- Why it matters
- Style slips are easy to miss and dull to point out, so they reach Peter, who points them out again.
- Closing it
- One shared check with a banned-word list, a readability limit and the Crank look written as rules: allowed colours, fonts and logo files. It lists every slip and where it is.
- Would live in
crank-tools, so every agent and every kind of draft uses the same rules (not built, no name yet).
Gap 11: A second-model critic as a standard step
- What’s missing
- Outside motion graphics, a draft can reach Peter checked only by the agent that made it.
- Why it matters
- An agent reviewing its own work shares the blind spots that made it. A different model, given the brief and the draft, reads it fresh.
- Closing it
- Make the critic a standard step before any draft reaches Peter: a different model reviews it against the brief and a short checklist, the agent fixes what it can, and the notes travel with the draft.
- Would live in
- Next to each skill that makes drafts, with its checklist and scored examples (gap 8) in git.
Today: a critic loop already runs for motion graphics. This gap is making it normal for every kind of draft.
Go deeper: craft gaps
- Plain English can be scored, roughly. Readability formulas such as Flesch–Kincaid estimate a reading grade from sentence and word length. A style check can set a limit and flag the worst sentences, and the critic or a person judges the rest.
- The Crank look, written as rules: exact colour values, named fonts, approved logo files. A program can check those in pages and slides. In film frames it needs the screenshot comparison from gap 3.
- Why a different model: an agent checking its own draft tends to share the blind spots that produced it. A second model, given only the brief, the draft and a checklist, comes to it fresh.
- The critic’s notes travel with the draft, so Peter sees what was flagged and what was fixed. The critic never sends, approves or merges anything.
All eleven, sorted
One question sorts every gap: could two careful people disagree about the right answer? The table after the diagram lists all eleven with where each would live.
Two of the tool gaps keep one call for a person. In gap 3, a person approves each new screenshot as the new baseline. In gap 6, a person decides which source is right when two disagree. The check itself still has one right answer.
The tool gaps split by where they’d live. Checks every agent needs (3, 4, 5, 7 and 10) belong in crank-tools. Checks tied to one repo’s code or data (1, 2, 6 and 9) belong in that repo. The judgement gaps (8 and 11) live next to the skill they test, with their examples in git.
| Gap | Group | Kind | Would live in | Today |
|---|---|---|---|---|
| Gap 1: CI checks on every PR | Verification | Tool | Each repo’s setup | Not built. No test checks on this repo’s PRs |
| Gap 2: A preview link for every PR | Verification | Tool | Each repo’s setup | Not built. PR #3’s preview: “Deployment Not Found” |
| Gap 3: Comparing screenshots to an approved version | Verification | Tool; a person approves new baselines | crank-tools |
Not built |
| Gap 4: Checks after a release goes live | Verification | Tool | crank-tools, plus each repo’s key pages |
Not built. Missing pages answer 200 |
| Gap 5: Env preflight | Verification | Tool | crank-tools |
Not built |
| Gap 6: Checks on incoming data | Quality | Tool; a person settles disagreements | Each data job’s repo | Not built |
| Gap 7: Tracing every figure to its source | Quality | Tool | crank-tools |
Not built |
| Gap 8: Scoring the model’s judgement | Quality | Judgement | Next to each skill, examples in git | Not built |
| Gap 9: Repeatable builds | Quality | Tool | Each repo’s setup | Not built. Versions float in this repo |
| Gap 10: Style rules a program checks | Craft | Tool | crank-tools |
Not built |
| Gap 11: A second-model critic as a standard step | Craft | Judgement | Next to each skill that makes drafts | Partly: motion graphics only |
Glossary
Every term on the page, A to Z. The tag shows which group it belongs to.
- Group VBaseline
- The approved screenshot that each new screenshot is compared with.
- BasicsBranch
- A separate line of work in a repo. Changes on a branch don’t touch the main code until they’re merged.
- BasicsCheck
- A test that runs on each piece of work and says pass or fail, so nobody has to remember to look.
- Group VCI
- Continuous integration: running a project’s build and tests automatically on every PR, and marking the PR as failing if they fail.
- Basicscrank-tools Crank practice
- Crank’s shared toolbox of commands, so every agent runs the same checks. None of the checks on this page is in it yet.
- Group CCritic
- A second model that reviews a draft against the brief and a checklist, and writes notes. It never approves or sends anything.
- Group VDeploy
- Putting a new version live, so people see it.
- BasicsDeterministic Crank practice
- Work with one right answer, so a program can check it and say pass or fail.
- BasicsDraft PR Crank practice
- A PR marked “not ready yet”. At Crank, agents’ PRs stay drafts until a person merges them.
- Group VEnv preflight
- A check, before a job starts, that the keys and settings it needs are there and the right shape. It never reads the secret values.
- Group VEnvironment variable
- A named setting a program reads when it starts, such as which account to use. Secret keys are often passed this way.
- Group QFresh
- Recent enough to trust. For a daily report, that means data from today.
- BasicsGap
- A mistake that can reach Peter, or a client, without anything catching it first.
- Group VHub-for-404
- When a missing page answers with the hub page and status 200 instead of an error, so it looks fine to a simple check.
- BasicsJudgement Crank practice
- Work that means weighing options, like choosing the wording. The model decides; scored examples and a critic show how well.
- Group QLockfile
- A file listing the exact version of every package a project uses, so every install gets the same ones.
- Group QPin
- To fix a tool at one exact version instead of taking whatever is newest.
- Group CPlain English
- Short sentences and everyday words, so a smart 14-year-old could follow.
- Group VPost-release check
- A check that opens the live page or service after a release and reads what comes back.
- BasicsPR (pull request)
- A proposed change that waits to be reviewed and merged.
- Group VPreview link
- A working copy of an unmerged change that you can open in a browser. Cloudflare Pages previews are public unless access is restricted.
- Group CReadability score
- A number that estimates how hard text is to read, from the length of its sentences and words.
- Group QRubric
- A short checklist for scoring work that takes judgement, so different scorers mark it the same way.
- Group QScored examples
- Examples with answers Peter has approved, kept in git and used to score the model’s calls after every change.
- Group VScreenshot comparison
- Capturing how a page, frame or slide looks and comparing it, pixel by pixel, with the approved baseline.
- Group VSecret
- A key or password that gives access to something. It never goes in a skill, a log, a commit or a chat.
- Group QSource data
- The original numbers a figure comes from, such as an ad platform’s report.
- Group VStatus code
- The number a web server sends with every page: 200 means OK, 404 means not found.
- Group CStyle check
- A program that flags banned words, hard-to-read sentences and off-brand colours or fonts in a draft.
- Group QTolerance
- How far apart two numbers can be and still count as agreeing, such as 1%.
- Group QTracing
- Finding every figure in a draft in its source data, or working it out again from source numbers.
Crank practice
Working decisions Peter and Cal have locked for Crank, and how each one shapes the gaps. These are Crank’s choices, not public teaching.
- Deterministic vs Judgement. If a check has one right answer, a program runs it and says pass or fail (gaps 1 to 7, 9 and 10). If it means weighing options, the model decides, and scored examples and a critic show how well (gaps 8 and 11).
- One shared toolbox. A check every agent needs goes in
crank-tools, so box agents and cloud agents run the same command. A check for one product lives in that product’s repo. - Draft, not merge. A passing check never merges anything. Agents open PRs as drafts, a person does every merge, and there’s no timed auto-merge.
- One official verify path for UI. pstack’s
/create-verification-skillwrites a product repo’s verification skill and Feature Map, and/maintain-verification-skillkeeps them current. Previews (gap 2) and screenshot checks (gap 3) feed that path. They don’t replace it. - Fix repeats at the strongest rung. Codebase > lint > rules > skills > style. When a critic keeps flagging the same thing, turn it into a check.
- Secrets stay out of skills. A preflight can test that a key is there and the right shape. Nothing reads, prints or stores the value.
- Nothing goes to a client without Peter. Checks and critics can flag, fix and draft. They never send.
Sources and credits
The gap list is Crank’s own. Anything marked “seen” was checked live on 7 October 2026. Platform behaviour comes from the providers’ docs, and quotes are copied from them.
- Crank practice and the gap list: Peter’s brief for this page, with the locked list of eleven gaps from Cal, Forge and miniForge in Crank Tools (the env preflight came from miniForge), and the note that a motion-graphic critic loop already runs.
- Agent Engineering Map, the companion page: Pillar C on verification, Pillar D’s correction ladder, and its Crank practice list.
- Live checks, 7 October 2026, run by the agent that built this page:
crank-experiments.pages.dev/e/no-such-page/andcrank-experiments.pages.dev/e/agent-engineering-map/both answered status 200 with the hub page (title “Experiments hub”).cursor-agent-engineering-map-d87a.crank-experiments.pages.dev, the preview address for PR #3’s branch, answered 404, “Deployment Not Found”.- The live site serves techniques 1 to 44, including #41 “Bot Jelly”. Only PR #1’s branch (
cursor/grokbot-demo-media-aa0b, still a draft) has 44 techniques.mainand every other branch have 40. - Origin reported no checks on PRs #1, #2, #4 and #5, and only Cursor Bugbot on PR #3.
- This repo’s
package.json:"node": ">=18", and a preview script that runsnpx --yes serve.
- Cloudflare docs:
- Serving Pages: with no top-level
404.html,Pages assumes that you are deploying a single-page application
andmatches all incoming paths to the root
. - Preview deployments:
By default, these deployment URLs are public.
An access policy can restrict them. - Pages Git integration: connects GitHub and GitLab repositories.
- Workers Builds: Cursor Origin integration: posts preview URLs in PR comments and creates check runs.
- Serving Pages: with no top-level
- pstack, by @poteto: the pstack plugin, whose planning playbook says
tests alone are not sufficient verification
, and its/create-verification-skilland/maintain-verification-skillskills.
How this page was made
Model: Claude Opus 5.5 at max effort (claude-opus-5-5-max, from this run’s metadata), in one Cursor cloud agent run with no subagents.
Token total: missing. The run metadata available to the agent doesn’t report a token count, so no total is shown and none was estimated.
Built 7 October 2026 from Peter’s brief, in the same run as the Crank Architecture page. Build note: src/e/verification-gaps/BUILD-NOTE.md.