Quality & Hardening

This page records how vfairness is tested and hardened before its first public release. Two kinds of work go into that.

Most of it is ordinary testing, and there is a lot of it: 10,398 tests across the metric, mitigation, LLM, XAI, monitoring, Pulse and rendering paths. They check that the library does what it says.

The rest is a hunt for one particular bug, and it is the reason this page is long. The bug happens when the library cannot work something out, quietly puts a harmless-looking number in its place, and then presents that number as a result. A 0.0 that reads as “no bias found”, when what really happened is that nothing could be measured at all.

Both are reported here with equal weight, and so is everything still open.

What the beta covers, and what it does not

The rule is what we told you to use. If a page on this site walks you into a capability, that capability is part of what the beta promises and has to be examined before the beta calls itself ready. It is not a judgement about which parts matter most; it is a reading of what has been advertised.

The rule cuts both ways, and that is the point of writing it down. Anything we are not prepared to examine has to stop being presented as ready, which is what the second row below is for.

In core: the assessment path itself (fairness metrics, bias detection, explainability); everything that hands you a verdict (report rendering, reporting, CI/CD gates, monitoring); every surface with a product page of its own (LLM testing, agent testing, and the pre-processing, in-processing and post-processing mitigation families); and the top-level namespace you import from.

In preview, each for a stated reason: the validity axis, which is deliberately unfinished until its reference set exists; the MCP tools, which you reach through another program rather than by writing vfairness code; the experimentation, causal and pulse operations; and the vision, networking and legal helpers, which are plumbing with no user-facing entry point. Preview means documented and usable, and not covered by the beta bar. It does not mean broken.

What it takes to ship. 498 core units are not yet clean: 0 nobody has examined and 0 carry a known open defect. That is the beta's remaining work, and it is the honest size of it. Every one of them is listed by name in the grid, filtered to core.

This boundary used to be three areas: fairness metrics, bias detection and explainability. It was drawn by functional area rather than by what reaches a reader, and two things fell through it. Report rendering draws the verdict you actually look at and is the worst-covered area in the library. CI/CD gates write a pass or a fail into a pipeline, which is the most dangerous place here for a value nobody measured. Both are advertised on the getting-started and sample-assessment pages, and neither was core.

Why hardening this library costs more than building it

A Python function has to return something. There is no natural way to say “I do not know” where a number is expected, 0.0 is the easiest thing to type, and it stops the program crashing. Every neutral value this programme has removed was written by somebody trying to be helpful.

Think of a smoke detector. Building one that beeps when you press the test button is easy. The hard part is proving it beeps when there is real smoke, and that it does not sit there silently when its battery is flat.

A fairness tool has the same shape. When it cannot measure bias and returns 0.0, that reads as “no bias found”. So:

• the failure is silent: nothing crashes, and no error appears
• the failure always points the reassuring way
• nobody reports it, because a customer told “your AI is fine” is a happy customer

That combination is why these defects survived so long. A bug that produces a wrong chart is reported in a day. A bug that produces a clean bill of health is never reported at all.

One function is one step to write and five to check

Writing a function is one step: make it work. Checking one honestly is five.

1. Build the situation where it cannot measure. Often harder than the function itself, because it needs a fitted model, a report object, or a populated store.
2. Run it and look at what actually comes back.
3. Judge whether that answer is honest. This is the expensive step, and a machine cannot do it.
4. Write a test that locks the honest behaviour in.
5. Deliberately break the code to prove the test notices. A test that cannot fail looks exactly like a test that passed.

Multiplied across the public surface, that ratio is the whole of this page.

What is tested and passing. The suite runs on every pull request and every push to main, on Python 3.11, 3.12 and 3.13. mypy src reports Success: no issues found in 228 source files and ruff check src tests reports All checks passed, both run 2026-09-18 and both blocking. Coverage is 81% of lines and branches against a floor of 55%, with a separate 80% floor on the lines a pull request adds. Eight workflows gate a pull request, among them a secret scan, a dependency scan, the export boundary and parity against reference libraries; six of the eight are path-filtered, so which of them fires depends on what the change touches. The capability census is complete: all 215 registered capabilities carry a grade, 158 proven and 57 semi-proven, and the 157 defects it found at the public entry points are all 157 fixed.

The second-round audit. The fix wave was audited again, and that audit recorded 59 claims. 59 are closed and 0 are still open (100% closed, as of 2026-10-02). A claim is closed when its defect is fixed and its test has been inverted into a pin, in the same commit. Open means the test still records the defective value the unit produces today, with the measurement in its docstring, so every open row is reproducible rather than suspected.

Generated from docs/bgl6-audit-register.json by scripts/bgl6_register.py, which reads the audit files themselves: each declares which of its tests still record an open defect, and a name that is not a test in that file is refused.

What is open, at the same weight. 618 grades have had no independent check, and 291 grades have been overturned when somebody did check, so that queue is the weakest claim on this page rather than a formality. 0 callables in the wider surface are carrying a proved defect that is not yet fixed, which is a change since 2026-09-18 and not a different way of counting. 1,336 of the 1,580 objects in the library carry no grade from the census; the wider-surface waves have reached 1,374 of them, and about the remainder this page claims nothing in either direction. Neither release bar passes: the beta gate stands at BETA READY, with 0 of its 8 criteria failing. Where it stands and what closes it are in the plan to publish.

Can I use this capability?

Everything else on this page describes what was built to harden the library. This section is the part you act on. Every public function, class and method carries one of three states, and the badge appears wherever the capability is documented: in the API reference, in its own docstring, and in the full grid at status of every capability.

Where the library stands, by scope

Not independently checked counts grades one examiner reached and nobody has yet tried to refute. Of the grades that HAVE been through that second examination, 291 of 872 (33%) were overturned, so a Checked unit in that column is a claim with evidence behind it, not a settled result.

Assembled by scripts/capability_status.py from evidence dated 2026-09-11, 2026-09-18, 2026-09-25, 2026-09-27, 2026-09-29, 2026-09-30, 2026-10-01. Core is what the beta promises: every area this project publishes a guide or a product page for, plus the top-level namespace. That is Agent fairness testing, Bias detection, CI/CD gates, Explainability (XAI), Fairness metrics, In-processing mitigation, LLM fairness testing, Monitoring, Post-processing mitigation, Preprocessing mitigation, Report rendering, Reporting and Top level. The rest is preview: documented and usable, and not covered by the beta bar.

• Checked means graded and assessed: executed on healthy input and on input where the thing it measures does not exist, and if it was found inventing a value, that was fixed. It is not a correctness certificate for the number it returns, and every row states its strength: held by a test, verified once with no test, or producing no verdict that could be fabricated at all.
• Fix pending means graded and assessed, and a defect is still open. Do not rely on it in the beta. Each row names which of three kinds it is, because they cost very different things to close: a fabrication still open, a fix an independent audit refused, or evidence too narrow to call it proven.
• Not checked means nobody has established anything either way. It is the absence of a measurement, not a failed one, and each row says why: never executed, unreachable without a hand-built fixture, called in a way that told us nothing, or executed with the evidence still unjudged.

Inside core sits the assessment path itself: fairness metrics, bias detection and explainability, 394 of the 1,580 code units. Of those, 394 have been examined and are clean, 0 carry a known open defect, and 0 have not been examined at all. Core is wider than the assessment path, and the wider figure is in what the beta covers.

Read those three numbers together. The examined figure is not a verdict on the path: it means that of the units somebody examined, none was found reporting a value it had not measured. The 0 nobody has examined are not evidence of anything in either direction, and collapsing them into the clean figure would be the exact defect this library is audited for, committed on this page about its own core. The 0 with a known open defect are counted here rather than left out: this paragraph said "none carries an open defect" until 2026-09-29, when the three figures were wired to the ledger that had been measuring them all along. Each unit says which of the three it is in the grid, and the core path can be listed on its own.

The scope is the point: the mitigation families, the LLM and agent testing surfaces and the operations tooling carry open defects of their own, and every one of them is in the grid.

Every public code unit has been executed and its result recorded. All 1,580 of them. A unit counts as examined when the capability census, a grading wave, one of the three probes or the test suite has run it, and the gate takes the union of those sources and names any unit missing from it. The evidence fails closed on the suite's own result: a run that was not green contributes nothing, because lines executed on the way to a failing assertion verified nothing.

Line coverage cannot see every kind of unit, and the gap is measured rather than assumed. A class body runs once when its module is imported, before coverage measurement begins, so no assertion can put an enumeration's lines into a coverage report. That split is measured separately, in a subprocess that starts coverage before the library is imported, and is deliberately not counted towards the execution criterion, because running a class body is not recording a result. Of the units coverage calls untouched, 132 as measured on 2026-09-25 and falling as the suite grows, 43 execute nowhere and 89 are class bodies. The split lands exactly on the class-versus-function boundary, which is what makes it credible: a class body runs on import and a function body does not.

The evidence behind each badge is checked, not taken on trust. Every graded row names the test that supports it, and scripts/verify_grade_attribution.py runs each named file and looks for the unit's own lines, because a row naming a test that never executes the unit is a claim with nothing behind it. It takes about forty minutes, so this is a dated snapshot and the date is part of the figure. Measured 2026-09-29: across 721 graded rows naming 108 test files it confirms 677 by line coverage, reports 39 as invisible to the measurement with the reason stated beside each, and finds 0 rows whose named test does not reach them. That last figure was three earlier the same day, and it had been published as zero: all three named a real test of the ADJACENT thing, which is why nobody noticed. They were corrected and this run is the confirmation.

Across the whole surface, no capability remains that anybody has SEEN fabricate. Every defect that was observed handing back a value nobody measured is closed: each was reproduced at the public entry point, pinned by a test, and each cause reverted separately to prove the test can fail. What is still open is 64 evidence gaps, 25 of them fixes an independent audit declined to certify and 39 where nothing was ever seen fabricating and the tests simply do not cover enough input to call the behaviour proven. That is a weaker worry than a known lie, and it is not the same as the criterion passing.

Two thirds of this surface was in the blind state when the programme started, and saying so is the reason this section exists. A library that publishes a percentage beside a scope nobody can see is committing the defect it was audited to remove.

The hardening programme

Every kind of testing is listed here with what it checks, how much of it there is, and where it stands today including what it leaves open. They are not ordered by how interesting they are. The fabricated-measurement audit, which the rest of this page goes into at length, is one row.

Where the two file counts come from, and why one of them is a range. 508 test files and 8,782 test function definitions are counts, re-run on every build: find tests -name "test_*.py" | wc -l and a grep for def test_. The split between the two largest groups is a sort, not a measurement, and the answer moves with the rule. Sorting by file name, 128 files carry a campaign prefix (audit, bgl_, surface_grade_), and 193 do if the readiness series and the explicitly honesty-named files are counted with them. Sorting by test-function name, 2,422 of the 8,782 test-function names carry the refusal vocabulary and 246 of the 508 files carry it in their file name. The word list is published in scripts/library_kpis.py as REFUSAL_VOCABULARY and REFUSAL_FILE_VOCABULARY and the counts come from that script, not from prose. The split is by NAME and it is a heuristic, so it says which tests are ABOUT honest refusal, not which of them are good.

What blocks a change, and what does not

Fourteen workflows sit in .github/workflows/, and they do not carry the same weight. A workflow that runs nightly is not a gate on a merge, so the two are separated here rather than presented as one list. Reproduce with ls .github/workflows/ and by reading the on: block of each file.

Run on a pull request

None of those eight marks a step continue-on-error, checked 2026-09-18 by grepping all eight files, so each one goes red on a real failure. Six of the eight are path-filtered, which is deliberate and is also a limit: a change that lands outside the filtered paths does not summon them at all.

Do not run on a pull request

The remaining three workflows handle worker deployment, documentation deployment and usage-map maintenance. They enforce nothing about the library, which is why they are described here rather than counted as gates. One of them does carry a pull_request: trigger: usage-map runs on a pull request and checks whether the usage maps have gone stale, but its pull-request step emits a ::warning and never fails, so nine workflows run on a pull request while eight can go red on one. Counted by trigger rather than by gate, the figure is nine.

What has been graded, and what has not

The audit above is the one with a census behind it, and this is that census. It counts one thing only: how much of the library has been put through the grading described further down. It is not a verdict on the other kinds of testing, all of which reach code this table calls not graded.

¹ A capability is not a fourth kind of object. It is one dispatch key in the registry, and it names one or more of the functions, classes and methods in the rows below it. The 215 rows resolve to 222 objects, which is why the graded column totals 222 and not 215: 98 functions, 117 classes and 7 methods. The capability row is not added into the total, so nothing here is counted twice.

This table counts the census only. An object in the not-graded column can still carry a grade from the second pass further down, which grades a different set by a different method. Not graded here means the census took no measurement. It is not a synonym for broken and it is not a synonym for fine.

A function is a module-level def in a non-private vfairness module. A class is a class defined in one. A method is a non-underscore method, counted at the class that defines it, so an inherited method is not counted once per subclass. The same four definitions produce the counts on the Getting Started page, so the two pages reconcile. Counted 2026-09-17 across 205 modules; capability grades measured 2026-09-11, surface grades 2026-09-18. Reproduce with python scripts/library_kpis.py.

The 215 capabilities

These are the calls that produce a fairness number or a verdict. Every one was run at its public entry, twice: once on data carrying a real difference, and once on data where the thing it measures does not exist. Each carries one of four grades.

This census found 157 defects at the public entry points. All 157 are fixed and independently re-checked, and 0 are open. They are a different set from the 242 findings of the deep audit in the record below, which counted across the whole codebase rather than at the public entry points.

These 215 and the 1,374 below are two separate measurements. They use different methods and different grade sets, so adding them together would produce a number that neither method produced.

The wider surface

The census covered the capabilities. A second set of waves covers the rest of the callable surface, 1,374 items so far, under five states rather than four. The fifth state exists because the census never needed it: out here, a callable can produce no fairness number and no verdict at all, and then the defect cannot apply to it.

756 of those 1,374 grades have been put to a second, independent check. 618 have not. They are reported as unaudited rather than counted as settled, and that gap is the largest single thing still open on this page. It is large because the most recent wave was also the biggest: it graded hundreds of callables in one pass, and auditing a grade costs about as much as producing it.

What is open

The 1,061 functions and methods the census did not grade sort by what they return. That sort is the shape of the work, not a verdict on any of them, and it is counted before the second pass takes any of them off the list:

Nothing takes a grade from that sort. A person or an agent has to run each one on input where the thing it measures does not exist, and read what comes back. The 275 ungraded classes are not in the triage, because a class does not return a measurement, its methods do. The named limits of this page, each with its reproduction, are in the open register.

How this was checked

The fix for the defect always has the same shape: three states, never two. Every surface that reports a result can say measured, failed, or could not check, and could not check is never quietly turned into one of the other two.

Every item is run twice. Once on healthy data that carries a real difference, so a working measurement has to find it. Once on data where the thing it measures does not exist: a single group, no positive outcomes, an empty table. A confident number coming back from the second run is the defect, and that is how each one on this page was caught.

A fix is not accepted on the word of whoever wrote it. A second, independent check runs against it with the job of proving it wrong. Where the two disagree, the second one is what gets published.

A test is not trusted until the bug is deliberately put back. After each fix the original defect is reinstated in the source, the named test is confirmed to go red, the healthy-data controls are confirmed to stay green, and the source is restored byte for byte. A test that stays green with the bug back in place proves nothing at all. On this project five guards turned out to be exactly that, green while the defect they existed to catch was live, and putting the bug back is what found them.

What the independent check actually does

The step above is done by whoever made the fix, so it cannot be the last step. Every grade is then handed to a second examiner with one instruction, prove this wrong, and with permission to run anything. That examiner is told what the grade claims and told, explicitly, that an audit which confirms everything is worthless and indistinguishable from an audit that did nothing.

It attacks five things in order, and the order is the measured one: most overturns come from the third.

1. Does the named test reach the unit? The test is run alone under coverage and the unit's BODY lines are read out, not its definition line, which executes on import. 26 grades in this repository named a test that never reached the code it was evidence for.
2. Can the guard be reached? A guard placed below a dispatch that has already chosen a different function cannot fire, however correct it looks.
3. Is there a SECOND input the test never tried? If the test covers an empty array, try one group. If it covers one group, try a group of one row. If it covers NaN, try infinity, or a threshold no data can breach. This is where most overturns come from.
4. Re-break the fix. Not the bug: the FIX. If the named test stays green, the fix is not load-bearing and the grade is wrong whatever the behaviour is.
5. Does the disclosure survive to a consumer? A correct "could not check" that a to_dict, a chart or a command-line envelope drops is still a defect, one layer up, and the producer-side test passes.

The examiner may return one of three verdicts, and the third is the reason the exercise is worth anything: confirmed, overturned, or could not check. A could not check is never recorded as a confirmation. Where it overturns, the grade published here is the second one, and the first is kept beside it.

What the checking found about itself

A page that reports only successes is not evidence of anything. These numbers say whether this method was able to disagree with itself.

Every wave, and how checked each one is

The totals above say how much was graded. They cannot say how evenly, and the difference matters: the two earliest waves were audited as they went and are fully checked, while the largest wave graded hundreds of callables in a single day and none of its grades has been argued with yet. A reader looking only at the merged total cannot tell those apart, so each wave is published with its own counts.

These rows do not sum to the totals above, on purpose. Each row counts its own wave's file, unmerged, so a callable graded by one wave and re-graded by a later one is counted in both. The totals above are the merged view, where the newest grade for each callable wins. Summing this table would double-count, and reconciling it by hand would hide which wave produced which grade, which is the one thing this table is for. “Pin sabotaged” counts rows carrying a record that the test was re-broken and shown to go red; the earliest wave predates that record being kept per row, so its figure understates what was done rather than describing it.

The clearest single case was the second wave. An entire audit pass was lost to a capacity limit. It was re-run, and 26 of the 129 grades that pass produced were overturned. That is the reason a grade nobody checked is reported as unaudited here rather than counted with the rest, and it is why the last column of that table is the column to read first.

Those two figures, 26 and 129, describe one pass inside the second wave. They are recorded in the repository, in the note on grading() in scripts/library_kpis.py, and not in the live statistics file the rest of this page reads, so they stay put when the page reloads.

Tests and coverage

Coverage is measured by CI, not recomputed when these pages are built. It is published as of a date and a commit, which is stated here rather than hidden in a tooltip.

What grading means

The word “graded” appears beside every number on this page and it carries no meaning on its own, so here is exactly what was done to each piece of code.

The defect being hunted. Sometimes the code cannot answer. The data has one group in it. Every score is missing. There are four relevant rows and no arrangement of them could ever reach significance. The defect is what happens next: instead of saying “I could not check this”, the code returns a neutral-looking number, and that number is then graded, counted, ranked and put in a report as though it had been measured. A 0.0 that reads as “no association found” when the statistic was undefined. An empty list that reads as “no proxies found” when the scan never ran. Each of those looks like a clean bill of health and none of them is one.

The fix is always three states, never two: measured, failed, or could not check. A caller has to be able to tell the third from the first without reading the source. The reverse counts too, and is worse: code that refuses when it genuinely could have measured something throws evidence away, and a detector that refuses everything passes every test about undefined input while finding nothing real. Both directions are checked.

Grading one piece of code, step by step

1. Read how it is really called. Its callers and the existing tests are the source for a realistic fixture.
2. Check the thresholds in its own signature first. Three times in this work a function looked honest when the test data was simply below a minimum group size the function itself declares. The refusal came from the fixture, not from the code.
3. Run it on healthy data carrying a real, findable difference, and confirm the measurement is exact. The expected value is recomputed independently rather than copied from what the code returned, because copying the output only proves the code agrees with itself.
4. Run it on data where the thing it measures does not exist. One group only. One outcome only. Every score missing. Zero rows. Every score identical. Two rows. One group with a single row. Text in an alphabet the code cannot read.
5. Run it with warnings switched on. A refusal carried only in a warning is invisible to anyone who suppresses warnings, and most callers do.
6. If it fabricates, fix it, then trace every caller. This is where the defect escapes: a caller comparing a not-a-number against a threshold silently gets “False”, and the fabrication reappears one layer up wearing a different shape.
7. Write a test that pins the behaviour, with a control asserting real data is still measured exactly.
8. Put the bug back on purpose and confirm the test fails. A test that cannot fail looks exactly like a test that passed. This step has caught more false confidence than every other step combined.
9. Hand the whole claim to a second, independent reviewer whose instruction is to prove it wrong.

Step nine is not a formality. 291 grades have been overturned at that step: better than one claim in five, each made after running the code, writing a test and sabotaging it, did not survive an independent check. That is also why 618 grades that have not reached step nine are published as unaudited rather than counted with the rest.

The five states

“Not a measurement” is the dangerous one, because it is the cheapest to assign and it closes the question. So it carries a rule: it may only be given after running the code and stating what it returns instead. Never from the name, never from the type, never from a guess. That rule exists because the guess was wrong repeatedly: 27 of 120 such dismissals were overturned, 15 of them to_dict methods. Counted from the grading record docs/surface-grading-2026-09-18.json on 2026-09-18: 120 items were first graded “not a measurement” and 27 of those grades did not survive the independent check. A to_dict looks like plumbing. It is the boundary where a careful three-state value is flattened for a caller, and where “could not check” quietly becomes a number.

A grade is not a guarantee. Proven means one specific thing: this code does not report a number it did not measure, and a test holds it to that. It does not mean the statistics are correct, that the method is the right one for your problem, or that the test covers every input. Those are different claims needing different evidence, and this library does not make them on the strength of a grade.

The plan to publish

A readiness claim without a plan behind it is an opinion, and this project has already published one of those. Two bars are measured by scripts/release_gate.py, which exits non-zero while anything is open.

This gate is not the test suite, and a failing criterion is not a failing test. The suite is 10,398 tests and it passes. The gate measures something else: how much of the public surface has been adversarially audited for one specific defect, and how much of it nobody has examined yet. It fails on coverage of the audit, not on broken behaviour. Read the two separately.

Where both bars stand. The beta gate returns BETA READY, with 0 of its 8 criteria failing. The answer at the top of this page lists each criterion, its number and whether it is met. This paragraph used to end “nothing is known to be wrong”, which stopped being true on 28 September: the second-round audit has 0 open records, every one reproducible today, and criterion B2b exists because the gate could not see them. Measured 2026-10-02; reproduce with python scripts/release_gate.py. Every figure in this section is read from that script, including the date.

How a release happens

The release pipeline has its own page: Release pipeline, with every step, who performs it, what stops a bad release, and the status of the first release.

What B5 defers to. The gate prints B5's measurement as see G5, which resolves in the gate's own console output and nowhere on this page, because the full gate's five criteria are not tabled here. G5 is the criterion B5 points at, and run 2026-09-18 python scripts/release_gate.py prints it as 0 open, 0 unproven: every one of the 215 registered capabilities carries a grade, none has an open defect and none is unproven. It is the single criterion either bar passes, and it covers the capability census only, not the wider surface in the row above it.

The denominator depends on which optional extras are installed. The census counts what it can import, so vfairness.mcp.server and the twelve functions it defines are visible to the count only when the optional mcp extra is present. A figure quoted from this page against a different environment will not reconcile. The denominator is a property of the measuring environment as well as of the library, and both scripts/library_kpis.py and the gate agree on it.

Why there are two bars. The full gate asks for every public code unit graded and no open defect anywhere. Measured, that is 904 remaining code units at 69,000 tokens of machine work each, so a release held to it is a release that never ships. A bar nobody can reach stops working as a bar. The beta bar is the smallest set of criteria under which publishing is still honest.

What a beta is allowed to be. It may ship with known defects. It may not ship with unknown ones, and it may not be quiet about the known ones. Criterion B2 decides the shape: a function returning a fairness number nobody measured can put a false clean bill in front of a user, while a renderer returning an empty chart cannot. Both are defects and only one blocks a beta. A code unit nobody has classified counts as the dangerous kind until someone says otherwise, because not having worked out what something returns is not evidence that it is harmless.

What it costs, from what was actually spent

Batching the audits cut their cost by 81 percent. Eleven audits in one reviewer produce the same verdicts as eleven reviewers, because each audit is a short self-contained check and the expensive part is start-up. That one change is the difference between this plan being affordable and not.

The strategy: stop paying review prices for machine work

A reviewer is needed to judge code, not to run it, and almost all of the measured cost was a reviewer building a fixture and calling a function nine times, which a program does for nothing. Three levers, in order of size: execute everything with a program rather than a reviewer; spend judgement only where that run raises a question; and batch every audit. The probe assigns no grades, deliberately. Its output is an observation, and turning that into “this fabricates” is judgement, because for some functions the neutral value on that input is the correct answer and refusing would destroy evidence.

Beta total: 15.6 to 20.4 million tokens. The probe has now run, and two of these estimates were wrong in opposite directions. The suspect count was guessed at roughly 300 from the rate at which earlier waves found fabrication; measured, it is 100, so phase 2 falls from 11 million to 3.7 million. Against that, the first version of this plan assumed the probe would close criterion B1 on its own. It does not: 167 code units could not be called without a hand-built fixture, so the probe says nothing about them. They are recorded as not reached, with the reason, and they count against B1 rather than being quietly dropped. That is phase 5 and it was missing here entirely. The two roughly cancel. The reach figures come from one run of scripts/surface_probe.py over 1,049 code units on nine input worlds, 882 reached and 167 not, recorded in docs/surface-probe.json and recounted from that file on 2026-09-18.

The suspect count of 79 could not be reproduced from that file. A suspect is defined here as a code unit that, on at least one world where the thing it measures does not exist, returned a neutral value and emitted no warning at all. Counting exactly that over docs/surface-probe.json on 2026-09-18 gives 100, not 79, so some narrowing is applied that is not written down here. Until that is run down, treat 79 as the figure phase 2 was costed on rather than as a settled count, and note that the larger figure would make phase 2 more expensive, not less.

What is uncertain, stated rather than hidden. The suspect rate in phase 2 was first taken from the waves so far, where 138 of 420 code units were found to be fabricating. The probe has since run and phase 2 is costed on its count instead, subject to the note above: on 2026-09-18 that count of 79 could not be reproduced from the probe's own file. The per-unit cost for probe-fed grading is still an estimate: execution was the bulk of the measured 53,000 and the probe removes it, but the first batch is what settles it.

The full record

Everything below was already on this page and none of it was removed. Some of it exists because something went wrong, and the record of that is the point. Every block here was re-read against the current measurement on 2026-09-18. Where a claim had moved since it was written, the correction is stated with its date beside the original rather than replacing it. Open any row to read it.

python scripts/library_kpis.py
python -m pytest --collect-only -q | tail -1
python -m pytest -q
ruff check src tests && ruff format --check src tests
mypy src
python -m build --sdist && ls -lh dist/

>>> report_card_to_svg(None)
'... COULD NOT CHECK: no gate decision was supplied, so nothing was
approved here ... DEPLOYMENT: NOT CHECKED ...'

>>> classification_fairness_report(y_true, y_pred, groups)   # one group never selected
demographic_parity_ratio == 0.0  ->  failed_metrics   (not passed_metrics)
demographic_parity_ratio == 1.0  ->  passed_metrics   (not failed_metrics)