Find a test object

Every fairness analysis needs something to test: a table of decisions, a model, a chat endpoint, an agent, a team of agents. This page tells you which object you need for the depth of analysis you want, and gives you a working one for every pathway, ready to download. Every file below was run through vfairness before it was published, and the page shows what happened.

What an assessment is, in 90 seconds

A fairness result is only as good as two things that must travel with it. Validant's article Precision Is Not Proof calls them the pointing and the seeing.

Three rules follow, and every object on this page is labelled with them in mind:

• The weakest of the three sets the ceiling. Deep access does not rescue a stale, unpinned model, and a careful method does not rescue outputs you were only told about. In the article's words: taken at your word is Indicative; decisions measured is Limited; plus confidence scores is Reasonable.
• A null result must say what it could have caught. "No disparity detected" is a finding only with its limiting magnitude: the smallest gap the test would have found, at a stated power.
• Not seen is not the same as seen and fine. If your tier cannot reach a question, the honest answer is "could not check", never a pass.

"Pointing says where she looked. Seeing says what could possibly have been resolved from there."

The access ladder

The tier is set by what the system gives back, not by how you connect to it. A chat window, a company's wrapper API and the provider's own API all return text, so all three are A1. Tiers and their ceilings, quoted from the article:

Fairness is an input-output discipline and does well at A1: group metrics need decisions, labels and group membership, never weights. Explainability is different: a claim about why the model behaves as it does starts at A3. Agent traces (which tool was called, where a case was handed off) are behaviour you watch, so they are A1 too: they widen what you see at A1 without reaching inside the model.

What data you need for what depth

One row per pathway. Each cell names the fields you need and what they unlock.

Predictive (tabular)

A predictive system turns a row of facts about a person into a decision or a score: approve a loan, invite to interview, flag for review. The test object is a table with one row per person: the decision, the group, and as much else as you can get. The same source can be walked up the ladder, which is what the Adult pack is for.

Our own synthetic files (lending, hiring, healthcare, recruitment) are in the catalogue. We measured what each contains before writing a word about it; the descriptions say what is in the data, not what was intended.

import pandas as pd
from vfairness import FairnessAnalyzer

URL = "https://vfairness.validant.ai/test-objects/data/"
df = pd.read_csv(URL + "predictive/adult_test_with_predictions.csv")

# A1 with labels: decisions, true outcomes, groups
a1 = FairnessAnalyzer(df.y_true, df.y_pred, df.sex).get_report(include_ci=False)
# A2: the same, plus the model's probabilities
a2 = FairnessAnalyzer(df.y_true, df.y_pred, df.sex, y_prob=df.y_prob).get_report(include_ci=False)
print(sorted(set(a2["metrics"]) - set(a1["metrics"])))
# ['auroc_parity', 'calibration_difference', 'integrated_calibration_index', 'multicalibration']

No labels at all? Then you have decisions and groups only, and run_pulse works in its label-free mode (FairnessAnalyzer needs y_true):

from vfairness.operations.pulse import run_pulse

decisions_only = df[["sex", "race", "age", "y_pred"]]
result = run_pulse(decisions_only, {"source_kind": "predictive", "protected_attributes": ["sex", "race"]})
print(result["data"]["verdict"]["headline"])

Synthetic: data with a known answer

Real data never tells you whether your tool is right, because nobody knows the true bias in it. Synthetic data does: you plant one mechanism and check that the tool finds that one and nothing else. The pack has a clean control and seven single-bias variants of the same 500-applicant recruitment funnel, each with its answer key in _bias_flags (drop it, and true_qualification_score, before you analyse). They are regenerated byte for byte by scripts/synth_single_bias_datasets.py.

One variant is a deliberate trap. surname_proxy.csv has no race column: surnames carry the signal and drive the decision. Nothing can make a direct claim about race from this file, because race is not in it. That is a declared ceiling, and a tool that reports race as "fine" here is claiming something it could not see.

This is what happened when we ran every variant through run_pulse as a US employment screen, with all ten protected attributes declared:

Tone alone is not detection. Graded on tone (any warn or critical), every file, the clean control included, would show six or more attributes "flagged", which says nothing.

Generative (LLM)

An LLM is tested by what it says. Change only the person in the prompt (a name, an age, a stated gender) and compare what comes back. The prompts can come from a benchmark; the answers can come from you running the model, or from someone who already did.

Where to get each tier: any chat endpoint is A1. Log probabilities (A2) come from OpenAI, vLLM, and Ollama from version 0.12.11; hosted routers such as Hugging Face Inference Providers and OpenRouter list the parameter, but whether it comes back depends on the model behind them, so check one response first. A3 means open weights you run yourself: GPT-2 is the cheapest full white box; Apertus 8B publishes its training data too, which makes it one of the few A4 subjects. vfairness's LLMApiProxy reads text only, so A2 analysis works the way the pack does: collect the probability yourself and analyse it as a score.

llm = pd.read_csv(URL + "llm/discrim_eval_qwen25_7b.csv")
measured = llm.dropna(subset=["p_yes"])            # A2: rows where p_yes could be read
print(len(llm) - len(measured), "rows could not be measured")
print(measured.groupby("race")["p_yes"].mean().round(3))
print(llm.groupby("race")["answer"].apply(lambda a: a.str.lower().str.startswith("yes").mean()).round(3))  # A1

Agent

An agent decides by acting: which tool it calls, whom it hands a case to, how many steps it takes. The test object is one row per episode with the group and the action, or the raw trace your framework already writes (OpenTelemetry, Langfuse, LangSmith). vfairness flattens raw traces into that table for you.

runs = pd.read_csv(URL + "agent/loan_agent_persona_runs.csv")
named = runs[runs.group != "control"]
result = run_pulse(named, {"source_kind": "agent", "protected_attributes": ["group"]})
agent = result["data"]["agent"]
print(agent["sampleAdequacy"]["perGroup"])   # episodes per group
print(agent["summary"])                       # what the probe measured

Print the probe's own summary, not only the top-level verdict. On this file the probe measures all six groups (120 episodes each) and finds no tool-selection bias, while the top-level verdict reads "Insufficient assessable data". That wording is wrong: it reports a measured null as a could-not-check. It is a known defect in how Pulse words a clean agent result, and the table below shows both readings until it is fixed.

Multi-agent

Several agents can each look fair and still be unfair together: one agent's note can steer the next. The test object needs each agent's output, not only the final one, so the question "is the system more biased than its parts?" can be answered.

committee = pd.read_csv(URL + "multi_agent/loan_committee_runs.csv")
named = committee[committee.group != "control"]
shift = (named.reviewer_after_risk - named.reviewer_alone_risk).groupby(named.persona_origin).mean()
print(shift.round(2))   # how far the screener's note moves the reviewer, by origin

Notebooks

Three notebooks walk through the objects above with every result executed, not described:

• notebooks/vfairness_10_test_objects_tour.ipynb: one source (Adult) up the ladder from A1 to A3, then the synthetic pack, planted against clean.
• notebooks/vfairness_11_llm_testing_demo.ipynb: discrim-eval at A1 and A2, and a live counterfactual test against your own endpoint.
• notebooks/vfairness_12_agent_testing_demo.ipynb: the loan agent with its noise floor, and the committee through the multi-agent analyzers.

What happened when we ran each file

Each object below was run through run_pulse by scripts/verify_test_objects.py. This is what the library returned, unedited. A verdict here describes the file, not a recommendation to use it.

Read the verdict as a triage screen, not as the size of the gap. Its line depends on the domain and jurisdiction you pass: in lending, severity needs a statistically confirmed gap with a meaningful effect size, and the four-fifths ratio (0.80) is shown as a guide. The lending file has real gaps (women approved 9.2 points less often, Black applicants 6.0 points less), but their ratios are 0.82 and 0.88, so it can read "within fairness budget". Read the measured description in the catalogue next to the tone, and decide your own threshold before you look.

Run on 2026-10-01.

Catalogue of downloads

Every file is listed in manifest.json with its sha256, size, source and licence. Check the hash after you download.

Classic public datasets, at their origin

Where a well-known dataset can be redistributed, we host a ready-to-run version above and link the origin here. Where it cannot, we link the origin only. Always cite the origin.

Trace formats your agent framework may already write: OpenTelemetry GenAI semantic conventions, OpenInference, Langfuse and LangSmith exports.

How these files were made

• scripts/build_test_objects.py builds the synthetic and Adult packs and writes the manifest. It refuses to build if the UCI origin file has changed.
• scripts/build_live_test_objects.py builds the LLM, agent and multi-agent packs against a local model and records its digest. discrim-eval is fetched at a pinned revision and checked by sha256.
• scripts/verify_test_objects.py runs every file through vfairness and writes the results shown above.

Names in the agent files are a standard correspondence-testing cue (Bertrand and Mullainathan, 2004). They signal a group to the model and say nothing about any real person. All personal details in the synthetic files are invented.