A Library on a Throughline

Writing down what already existed

The first day produced the description. It is not a summary. It says which four things the build compiles into the worker, what the policy in the page allows and forbids, which errors each operation can raise and by what name, what happens when a clone is interrupted, and what the product must refuse. 240 of the 664 statements are tests.

Then a challenge pass ran over it. Ten challengers went at every family of statements, each answered by a sceptic whose job was to refute the challenge. 118 findings survived that and were applied. Twenty-nine questions could not be settled by argument and were written down for the person to answer.

None of this had found anything wrong with the product yet. It had found things wrong with the description.

Seven things that were already wrong

These came out over the following two days, as the statements were read against the standards they claim to follow and against each other. Each one existed in running code, or in a record somebody had already signed. None was introduced by the work.

A security policy weaker than the standard it follows

The product sends a policy to the browser saying where code and connections may come from. One line of it was weaker than the security standard the product follows requires, and the stronger setting would have cost nothing at all, because the product never uses the thing the weaker setting permits.

Nothing failed. It came out of reading the standard's clause and the product's own statement side by side. The same policy is live in the editor today, and this is one of the two faults still open.

A sign-in relay that would have handed over a secret

Signing in goes through a small piece of code that passes a request to the code-hosting service, carrying a secret that proves which application is asking. It followed redirects automatically. A redirect answer would have carried that secret to whatever address the redirect named.

In the browser, the policy above would have limited the damage. This part does not run in a browser, and nothing limits it there. At the moment this was found, the word "redirect" appeared once in 658 statements, in an unrelated sense. There was no test for it, because nobody had ever said it mattered.

A sign-in relay anyone could switch off

The same relay checks who is calling by reading a header the browser sets. The statement describing that check says, in its own reasoning, that the header is the only control. A header is trivial to forge from a command line.

So anybody who knew the relay's address could have used up the application's sign-in allowance at the hosting service, and nobody else could have signed in until it reset. A second statement says the relay keeps no records, which is a good property and means nothing afterwards would have shown it had happened.

A rule already broken twenty-eight times

The estate keeps a standing rule for its own work: one statement per behaviour, and one test per statement. In this graph, 26 statements were checked by two tests and 2 by three. Nothing flagged it, because the automated check asks only whether a statement has a test and never counts how many.

The rule turned out to be the thing that was wrong. One test is a floor, not a ceiling. It was rewritten and signed again. Two details are worth keeping: the wording was noticed first and the twenty-eight counted afterwards, and the wording had come from a note in the AI's own memory rather than from the person whose rule it was.

A number repeated in nine places, wrong in all nine

Nine statements said the product installs four particular components. The fourth had stopped existing weeks earlier, when the tool it came from absorbed it.

The first correction pass matched two of the three ways the number had been phrased and missed the third, so it fixed three statements and left six. Two of those six were already signed, and correcting them cost two more signatures. The number has now been removed rather than changed to three, because a count written into a title buys a fresh signature every time the count moves.

The failure is ordinary: a fact repeated in nine places eventually disagrees with itself. What is not ordinary is that the disagreement could be found, because all nine places said it out loud.

Sixty-one claims of conformance that stopped being made

The editor's older statements carry 63 claims to meet particular clauses of a security standard, spread across 21 of them. Twenty of those 21 were carried into the new description. The new description did not yet declare the standard, so the clauses had nothing to attach to, and 61 of the 63 claims quietly stopped being made. No statement recorded the loss.

Which claims they were is the sharp part. Three of the editor's statements exist only because of an audit a month earlier into false claims of conformance. One of them exists because a clause about encrypting traffic had been claimed by six statements and delivered by none of them: five decide who may sit in the network path and not one mentions encryption, and the word HTTPS appeared exactly once in the whole of the older graph. A conforming implementation could have used unencrypted connections throughout.

The corrections survived the move. The clauses that forced them did not. The standard is declared again now, and reattaching the individual claims is the second of the two faults still open.

One reason standing in for two decisions

A decision not to keep a register of threats gave as its reason a sentence that belongs to a different decision entirely — the decision not to adopt the security standard at all, which carries the same sentence.

Neither the person nor the AI noticed until he had signed it and withdrawn the signature a few minutes later. His second objection was to the text, which claimed that threats can only be assessed against a finished deployment. The two relay faults above had disproved that earlier the same day, at the library's own boundaries, with nothing deployed. A third threat, that an issued key cannot be withdrawn, had no home at all. Both decisions are now overturned, and the library borrows from a published threat-modelling method instead.

Signing it

Nothing the AI writes counts until the person signs it, and there were 664 statements to sign.

He signed 25 on the evening of the first day, then 118 more in one sitting: a block of guarantees and the tests that check them. Two days later, on the morning of the third, 473 were still waiting. That afternoon he signed 280 in sixteen minutes — every statement that was not a test — and then worked through the tests area by area over the following hour, in blocks of 44, 21, 19, 18, 16, 15, 13, 13 and 18. The last block was signed 53 hours after the first statement was written.

Sixteen minutes for 280 items is three and a half seconds each, and that is not reading. The honest description is that it records decisions mostly taken earlier, in the batches and the twenty-nine questions that came before it, and that some of them were taken on trust. The tests were deliberately held out of that run and gone through separately, which is the part that was read.

There was a cost the tooling imposed as well. In the middle of his run the AI amended six statements. Five signatures he had given minutes earlier stopped being valid and had to be given again. The tool reported it, which is the only reason anyone knows. Over the whole three days, 81 statements had to be signed a second time.

The obvious question is what those sixteen minutes do to the claim the method rests on, that nothing counts until a named person has signed it. Three and a half seconds is not agreement in any sense a reader would accept, and for those 280 the signature records that he accepted the AI's account of decisions he had mostly settled earlier, not that he read each one.

The unexpected part is that careful signing would not have caught these seven either. The first of them is the clearest case. The statement carrying that security policy was signed on the first evening, in a sitting of 118 that took an hour and three quarters — about fifty-four seconds an item, and he was reading. The fault in it surfaced thirty-five hours later, and not from anyone re-reading it: it surfaced when the statement was put next to the clause of the standard it answers. None of the seven was found by looking harder at one statement. Each was found by putting two things side by side — a statement and a clause, a statement and its own reasoning, a statement and the eight others saying the same thing. That is a different activity from signing, and the queue is not what pays for it.