From question and answer to evidence-backed developer playbooks

I lost most of an afternoon to an answer with several thousand upvotes. It was clear, well written, confidently accepted, and it did not work. It had worked — for the person who wrote it, on a version they never named, in an environment nobody recorded. None of that was visible from the page.

The failure there is not dishonesty. Nobody did anything wrong. The failure is structural: the format has nowhere to put the information that would have saved me. DEV.ITISYOU is an attempt to build a format that does.

What a question-and-answer page cannot record

A thread is a snapshot of a conversation. It records that someone had a problem, that someone else proposed a fix, and that a number of people approved. What it cannot express, because there is no field for it:

• Which versions the fix was tested against, as data rather than prose.
• Who reproduced it, on what, and when it last actually worked.
• That it worked on one runtime and failed on another — the two reports simply sit next to each other as disagreement.
• That it stopped working eighteen months ago. Votes accumulate; they do not expire.

The last one is the sharpest. A vote count is a measure of how many people approved at some point in the past, presented in the interface as a measure of how likely something is to work now. Those are very different quantities wearing the same badge.

The shape of a playbook

A playbook is not an answer. It is a diagnostic path. You paste the error exactly as you received it, and exact error codes and stack frames are matched first, then narrowed by your versions.

From there you take one diagnostic test at a time and report what you actually observed — including “I can’t tell”, which is a real option and matters more than it sounds. In practice a large share of debugging time is spent on a step whose result the person could not determine, quietly guessing rather than admitting it. Making that a first-class answer means the path can branch on uncertainty instead of pretending it away.

Every fix then shows who reproduced it, on which versions, and when it last worked. Failed attempts are shown as well, which is the part people find strangest.

Three decisions that carry the design

Evidence sticks to the revision, not the page. Editing a playbook creates a new revision, and the old evidence stays attached to the text it actually tested. This is the single most important rule in the system. On a conventional wiki, evidence gathered against one version of a page silently transfers to whatever that page later becomes — which means a well-reproduced fix can be edited into something nobody has ever tested while keeping every signal that made it trustworthy.

There is no verified tick. Confidence is derived from reproduction records and always displays the counts behind it. A badge compresses a distribution into a symbol and throws away the part you needed. “Worked for 3 of 4 reports on this version” and “worked for 340 of 400” deserve to look different, and under a tick they do not.

Failures are kept. A fix that failed on a particular runtime version is recorded as exactly that, rather than being buried in a downvote. Disagreement is segmented by environment, because two engineers on different versions reporting opposite results are usually both telling the truth. A voting system reads that as a dispute to be resolved. It is not a dispute. It is two facts.

What the site refuses to do

Dev publishes a list of things it will not do, and each is a real constraint:

• It will not run code for you, or from you. That rules out automated reproduction, which would be genuinely useful, and rules out a large class of security problems along with it.
• AI does not get to decide that a fix works. AI can help structure a playbook. It cannot supply the evidence, because the evidence is the entire product.
• No success percentage from three reports. A percentage implies a sample. Three reports are not a sample.
• No sign-in to read or search. A knowledge base behind a login is a lead-generation form wearing a library’s clothes.
• No ranking by reputation points. Reputation measures participation. It does not measure whether this fix applies to your version.
• Nothing hidden when it stops working. A fix that has stopped working is one of the most valuable records in the system.

What this costs

Every one of those decisions makes the library grow more slowly, and I would rather say that plainly than pretend the trade-off is free.

Coverage is narrow. The areas with real depth today are Cloudflare Workers, Cloudflare D1, PostgreSQL, Docker, Kubernetes and Node.js. Arrive with a problem outside those and the honest description of what you will find is: nothing. Narrow and deep beats broad and thin right up until the moment it looks empty, and I do not have a way around that except time.

There is a harder unknown too. This model asks contributors to have their failed fixes recorded permanently, attached to their name, as durable evidence. I think that is right — a failure on a stated version is data, not a mistake. Whether contributors will accept it at any real scale is genuinely untested, and it is the assumption most likely to be wrong.

Dev is free to read, with no account required, and limited advertising is planned but not running. Which means it currently costs money to operate and earns none — a fact I would rather state than leave for someone to work out.