Quality & Hardening

How vfairness is tested, audited, secured, and hardened for its first public beta.

Before its first public beta on PyPI, vfairness went through a structured hardening program rather than a single cleanup pass: a multi-agent deep audit of the whole codebase, a full static type-checking burndown, a battery of correctness and honesty tests, a supply-chain and security posture built into CI, and packaging and release-integrity checks. This page inventories every measure. Each item carries its internal issue id (VB-*) so it maps back to the changelog.

The guiding principle

Honesty over optimism. The library should never silently pass, silently drop a small group, or report a number it did not actually compute. Most of the measures below exist to enforce that one rule. The beta exit criteria live in the Getting Started guide and the beta programme notes; this page is the detailed companion.

At a Glance

Headline figures, as of August 2026. Live values come from the release manifest and CI, not from this page.

1,900+

automated tests, on Python 3.11 / 3.12 / 3.13

mypy errors (blocking), plus a strict island on the core

clean

ruff lint + format, both blocking gates

242

deep-audit findings fixed, zero open

98%

mutation score on the boundary core

green

security scanning (Bandit + pip-audit), blocking on every push

gated

trusted publishing, provenance, SBOM, clean-room wheel install + public-API smoke: all release-blocking; the first pipeline publish (0.1.0) is still pending

Correctness & Scientific Validity

The metric core is checked against ground truth from several independent directions, not just against its own past output.

Reference-library parity VB-TEST-4

Metrics are cross-checked against established libraries, and every intentional difference is documented in DIVERGENCES.md rather than hidden.

Synthetic analytic oracle VB-TEST-1

Known-answer checks on constructed data where the correct fairness value is derivable by hand.

Property-based tests VB-TEST-2

Hypothesis generates adversarial inputs to probe invariants that fixed examples miss.

Impossibility relationships VB-TEST-3

Tests assert the known mathematical tensions between fairness criteria hold, and correctly vanish at perfect accuracy.

Determinism VB-TEST-5

Repeated runs on the same input produce identical output; bootstrap paths take an explicit random_state.

Insufficient-evidence honesty VB-API-4

No code path silently passes or drops a small group. Strata below the group-size gate return an explicit insufficient-evidence verdict via assessment.insufficient_evidence_groups.

Undefined metrics are not failures VB-EVAL-1

A metric that cannot be computed (returns NaN, e.g. an error-rate metric when a group has no positive labels, or R² when a group has constant y_true) is surfaced in assessment.not_assessable_metrics and excluded from the score, never reported as a fairness violation.

No silent swallow VB-SEC-4

An AST-level guard test forbids the metric core from swallowing computation errors across evaluation, in-processing, and post-processing.

The Deep-Audit Campaign

A two-workflow, multi-agent deep audit (one static, one empirical, each adversarially verified) examined the whole library and produced 242 findings. All were fixed across five waves to zero open items; the suite grew from 831 to more than 1,280 tests in the process.

Representative fixes

Statistics honesty. fisher_exact_test uses the real Fisher exact test (no silent chi-square degrade above n=200); Bayesian group intervals report the observed rate as the point estimate; a minimum-detectable-effect is computed from the requested alpha and power.
Calibration correctness. BetaCalibrator is real Kull et al. 2017 beta calibration rather than a renamed Platt transform; group calibrators pass unknown group ids through untouched and disclose the mismatch instead of silently zeroing them.
Threshold optimization. The optimizer that previously crashed on fit now searches thresholds jointly by coordinate descent and validates the final assignment.
Falsy-zero bugs. A family of bugs where a legitimate 0 (a zero-tolerance threshold, a fairness score of 0, an explicit 0.0 alert bound) was treated as absent were fixed across CI/CD, monitoring, and rendering.
Rendering robustness. Hostile inputs (empty modules, None nests, stringified numbers, infinities) either render or raise a clear ValueError naming the template, instead of leaking a ZeroDivision or overflow error.

Code-Quality Gates

Lint gate VB-LINT-1

ruff check src tests runs on every library change and is blocking. The tree is clean under the E/F/I/N/W rule set.

Format gate VB-LINT-2

The whole src and tests tree is ruff format-clean (double quotes, space indent) and ruff format --check is blocking. The one-time reformat commit is recorded in .git-blame-ignore-revs so git blame skips it.

Type gate VB-TYPE-1

A full type-error burndown took mypy src from 500 errors to zero across 82 modules with root-cause fixes, and mypy src is now blocking. The pass also surfaced and fixed real bugs, such as a subgroup-robustness audit and a correlation-based proxy detector that were each silently dead because a value was passed in the wrong argument position. The package ships a py.typed marker (VB-API-1) so downstream projects type-check against the library's own hints.

Strict type island VB-TYPE-2

Foundational public modules and core helpers are additionally held to mypy --strict via a per-module override; the island expands outward as more modules are cleaned.

Public API-surface snapshot VB-API-3

A test freezes the public surface, so an accidental breaking change fails CI.

Mutation testing VB-TEST-9

mutmut verifies the tests actually catch injected faults; the boundary core scores 57 of 58 mutants killed.

The Test Suite

More than 1,900 tests run on the full supported Python range in CI. Beyond the correctness checks above, the suite guards output stability, scale behavior, and the library's runtime guarantees.

Branch coverage and new-code gating VB-TEST. Coverage counts untaken branches, not just unexecuted lines; the CI floor ratchets up as the suite grows, and a pull-request diff-cover gate requires the lines a change adds to be at least 80% covered.
Executable documentation VB-DOC-3. Snippets in docs/examples.md run in the suite via doctest, so a documented example that breaks against the real API fails the build.
Report snapshots VB-TEST-10. Rendered reports are pinned so a change in output is deliberate, not accidental.
Performance and scale VB-PERF-1. Large-input behavior is exercised (and gated behind a slow marker so a fast run can skip it).
Zero-telemetry VB-GOV-2. A test asserts no network access happens during metric computation.
Deserialization safety. A scan asserts the absence of unsafe loaders, including aliased imports.

Security & Supply Chain

Static and dependency scanning VB-SEC-4/8

Bandit (source) and pip-audit (known vulnerabilities, run in strict mode against a non-editable pip freeze) run on pull requests and on a weekly schedule. A CodeQL job is wired up but stays skipped while the repository is private, because code scanning needs GitHub Advanced Security; it re-arms automatically if the repository goes public.

Supply-chain posture VB-SEC-5/7

Dependabot updates and a CycloneDX SBOM generated at release. An OpenSSF Scorecard workflow is configured but stays skipped while the repository is private.

Deserialization safety VB-SEC-6

The xai sidecar refuses to deserialize untrusted model payloads unless the caller explicitly opts in, closing a remote-code-execution vector.

Second security-audit hardening (defense in depth)

A follow-up pass added, with no known exploit: the MCP load_dataframe tool rejects remote data_path URLs so an agent-supplied path cannot become network egress or an SSRF fetch; the Pulse artifact download pins the scheme to https and caps response size; the deserialization scan resolves aliased imports and covers marshal, pandas.read_pickle, jsonpickle, and shelve; error logs no longer echo sensitive payloads; and CI workflows declare least-privilege (contents: read) token scope.

Packaging & Release Integrity

Licensing VB-LIC-1 / VB-PKG-1. Apache-2.0 with an accompanying NOTICE and a single-source version.
Lean source distribution VB-PKG-2. An sdist allowlist drops the archive from roughly 256 MB to about 1 MB.
Build verification. Every release runs twine check and check-wheel-contents on the built artifacts.
Clean-room install smoke test VB-REL-3. Before any publish, the built wheel is installed into a fresh environment and its public API is exercised from outside the source tree, so a missing data file, wrong entry point, unshipped subpackage, or missing dependency blocks the release.
Trusted publishing VB-REL-1/2. Tag-triggered release to TestPyPI, then PyPI, both via PyPI Trusted Publishing (OIDC, no stored token) with provenance attestations, and a required human approval before the production publish.

API Stability & Methodology

Frozen public surface VB-API-2. The surface listed in API_STABILITY.md is already treated as stable: it changes only after a deprecation warning, and removals are reserved for 1.0.0.
Public exception hierarchy VB-API-5. Errors are rooted at VfairnessError while still subclassing the built-in they replace, so existing handlers keep working.
Methodology versioning VB-DOC-1. Every FairnessAnalyzer.get_report() stamps a methodology_version, so a rating records the methodology that produced it independently of the code version.
Canonical module taxonomy VB-PKG-3. The public surface is 15 top-level sub-packages (6 pipeline, 8 specialized, 1 infrastructure), each declaring __all__ and enforced by a test.

Continuous Enforcement

The gates above are not one-time cleanups; they run in CI on every change.

Workflow	What it enforces
Full test suite	The whole suite with coverage on Python 3.11 / 3.12 / 3.13
Quality	`ruff check`, `ruff format --check`, and `mypy src`, all blocking; a PR-only diff-cover check
Security	Bandit and pip-audit (CodeQL skipped while the repository is private)
Scorecard	OpenSSF supply-chain posture (skipped while the repository is private)
Mutation testing	`mutmut` on the boundary core, weekly and on demand, informational rather than blocking
Cross-library parity	Metric agreement with reference libraries
Pulse contract	The Pulse assessment output contract
Changelog check	A changelog entry accompanies changes
Release	Build, verify, clean-room wheel-install smoke test, SBOM, and trusted publishing; tag-triggered, awaiting its first tagged release

The Beta Promise

Beta means ready for real users to rely on while the library keeps improving. It is free and open source under Apache-2.0 and stays that way.

What beta does not promise

vfairness is a community project with no service-level agreement: no guaranteed response time, no uptime commitment, and no contractual assurance. APIs may still change before the stable 1.0.0, though the frozen surface above is already treated as stable. Managed, independent, service-backed assurance lives on the validant.ai platform, not in this library.