Agent Fairness Testing

Test AI agents for bias in tool selection, retrieval, actions, and delegation.

Why Agent Testing?

Agents don't just generate text — they take actions, use tools, and make decisions. Bias can enter at every stage of the pipeline: which tool the agent selects, what documents it retrieves, how it reasons through a chain of thought, and what actions it ultimately takes.

No existing framework covers this. Traditional fairness toolkits measure model outputs — predictions, scores, classifications. But an agent that routes loan applications to different review queues based on applicant demographics, or retrieves different precedent documents for different ethnic groups, will pass every traditional fairness test while producing discriminatory outcomes.

vfairness is the first framework to provide structured fairness testing for agentic AI systems.

Each type maps to a distinct stage in the agent pipeline, requiring its own detection strategy.

Run a correspondence test on an agent and track its full pipeline for bias.

from vfairness.agents import CorrespondenceTester, PipelineTracker

# 1. Correspondence testing — same query, different demographics
tester = CorrespondenceTester(
    agent_fn=my_agent.run,
    protected_attributes=["gender", "ethnicity"],
    n_iterations=200
)

# Generate matched pairs and measure outcome differences
results = tester.run(
    base_prompts=[
        "Review this loan application: {profile}",
        "Evaluate this candidate: {profile}",
    ],
    demographic_profiles=tester.load_profiles("hiring_v2")
)

print(f"Demographic Parity Gap: {results.dp_gap:.3f}")
print(f"Action Disparity:       {results.action_disparity:.3f}")
print(f"Statistically Significant: {results.is_significant}")

# 2. Pipeline tracking — instrument every stage
tracker = PipelineTracker(agent=my_agent)

with tracker.trace() as trace:
    result = my_agent.run("Process application #4821")

# Inspect per-stage bias
for stage in trace.stages:
    print(f"  {stage.name}: bias_score={stage.bias_score:.3f}")

What you can test depends on how much access you have to the agent system.

Agent bias is not static. As knowledge bases update, tool configurations change, and user populations shift, bias can emerge or disappear over time. vfairness provides two statistical process control methods for continuous monitoring.

CUSUM (Cumulative Sum)

Detects sustained shifts in bias metrics. CUSUM accumulates deviations from a target value and signals when the cumulative sum exceeds a threshold. Best for detecting gradual, persistent drift.

EWMA (Exponentially Weighted Moving Average)

Smooths recent observations with exponential decay, making it sensitive to small but consistent changes. The smoothing parameter λ controls the trade-off between responsiveness and false alarm rate.

from vfairness.agents import DriftMonitor

monitor = DriftMonitor(
    metrics=["dp_gap", "action_disparity"],
    method="cusum",          # or "ewma"
    threshold=0.05,          # alert threshold
    warmup_periods=50        # baseline calibration window
)

# Feed observations from production
for batch in daily_agent_logs:
    result = monitor.update(batch)
    if result.alert:
        print(f"DRIFT DETECTED at {batch.date}: "
              f"{result.metric} shifted by {result.magnitude:.3f}")
        print(f"  CUSUM statistic: {result.cusum_value:.3f}")
        print(f"  Recommended action: {result.recommendation}")

# Visualize drift over time
monitor.plot(output="drift_report.svg")

All agent testing results include production-ready features for audit compliance and system integration.

Audit Trail & Serialization

Every result carries RunMetadata with timestamp, library version, and full parameter snapshot:

from vfairness.agents import CorrespondenceTester

tester = CorrespondenceTester(alpha=0.05)
result = tester.analyze_outcomes(outcomes_a, outcomes_b, artifact_type="resume")

# Audit metadata on every result
print(result.metadata.timestamp)        # ISO 8601 UTC
print(result.metadata.library_version)  # "0.1.0"
print(result.metadata.parameters)       # {"alpha": 0.05, "artifact_type": "resume", ...}

# Serialize for database storage or reporting
data = result.to_dict()    # Plain dict
json_str = result.to_json()  # JSON string

Structured Logging

All operations emit structured logs via Python's logging module:

import logging
logging.basicConfig(level=logging.INFO)

# Operations now emit structured logs:
# INFO:vfairness.agents.correspondence:Starting correspondence test with 50 samples per group
# INFO:vfairness.agents.tool_bias:Analyzing 3 tool types across 100 traces
# WARNING:vfairness.agents.pipeline:Sample size (12) below recommended minimum of 25