Agent Fairness Testing

Test AI agents for bias in tool selection, retrieval, actions, and delegation.

Why Agent Testing?

Agents don't just generate text — they take actions, use tools, and make decisions. Bias can enter at every stage of the pipeline: which tool the agent selects, what documents it retrieves, how it reasons through a chain of thought, and what actions it ultimately takes.

Traditional fairness toolkits measure model outputs — predictions, scores, classifications. But an agent that routes loan applications to different review queues based on applicant demographics, or retrieves different precedent documents for different ethnic groups, will pass every traditional fairness test while producing discriminatory outcomes.

vfairness provides structured fairness testing purpose-built for agentic AI systems, covering stages that traditional toolkits do not measure.

The 7 Agent-Specific Bias Types

Each type maps to a distinct stage in the agent pipeline, requiring its own detection strategy.

Tool Selection Bias ToolBiasAuditor

Agent picks different tools based on demographics. A hiring agent might use a rigorous skill-test tool for one group and a simpler keyword matcher for another.

RAG Bias RAGBiasAnalyzer

Retrieval amplifies biases from documents. If the knowledge base over-represents certain demographics, the agent's retrieved context — and therefore its decisions — will be skewed.

Reasoning Chain Bias ReasoningChainAnalyzer Roadmap

Chain-of-thought doesn't reflect actual decision. The agent's stated reasoning may be a post-hoc rationalization that masks demographic-dependent decision paths.

Multi-Agent Amplification CompositionalityAnalyzer

System bias does not equal component bias. Two individually fair agents can produce unfair outcomes when composed, due to correlated errors and cascading thresholds.

Delegation Bias DelegationRoutingAuditor

Orchestrator routes differently by demographic. A triage agent might escalate cases from one group to senior review while auto-resolving equivalent cases from another.

Action Bias ActionBiasAnalyzer

Different tangible outcomes by group. The agent approves, denies, or modifies real-world actions (salary offers, credit limits, appointment slots) differently across demographics.

Feedback Loop Bias FeedbackLoopDetector Roadmap

Agent reshapes its own data landscape. Biased actions create biased training data, which reinforces the original bias in a self-perpetuating cycle.

Quick Start

Run a correspondence test on an agent and track its full pipeline for bias.

python

from vfairness.agents import CorrespondenceTester, PipelineTracker

# 1. Correspondence testing: same artifact, different demographics
tester = CorrespondenceTester(alpha=0.05)
pairs = tester.create_paired_artifacts(base_artifact, "gender", ["male", "female"])

# Submit the pairs to the agent, then analyse the outcomes it returned
result = tester.analyze_outcomes(outcomes_a, outcomes_b, artifact_type="resume")
print(f"Disparity: {result.disparity_metric:.3f}  p={result.p_value:.4f}")
print(f"Statistically significant: {result.is_significant}")

# 2. Pipeline tracking: record every stage, then locate the source
tracker = PipelineTracker(stages=["retrieval", "reasoning", "action"])
tracker.record_stage("retrieval", outcomes_a_retrieval, outcomes_b_retrieval)
tracker.record_stage("reasoning", outcomes_a_reasoning, outcomes_b_reasoning)
tracker.record_stage("action", outcomes_a_action, outcomes_b_action)

for stage in tracker.compute_cumulative():
    print(f"  {stage.stage_name}: cumulative={stage.cumulative_bias:.3f} "
          f"contribution={stage.stage_contribution:.3f}")
print(f"Primary bias source: {tracker.identify_bias_source()}")

Access Tiers

What you can test depends on how much access you have to the agent system.

Test	UI Only	Submission API	API + Traces
Manual correspondence	✓	✓	✓
Automated correspondence	—	✓	✓
Tool selection audit	—	—	✓
Multi-stage tracking	—	—	✓

UI-Only Testing Has Limits Limitation

With UI-only access, you can only perform manual correspondence tests. You cannot detect tool selection bias, reasoning chain bias, or delegation bias without API-level access to the agent's internal traces.

Drift Detection

Agent bias is not static. As knowledge bases update, tool configurations change, and user populations shift, bias can emerge or disappear over time. vfairness provides two statistical process control methods for continuous monitoring.

CUSUM (Cumulative Sum)

Detects sustained shifts in bias metrics. CUSUM accumulates deviations from a target value and signals when the cumulative sum exceeds a threshold. Best for detecting gradual, persistent drift.

EWMA (Exponentially Weighted Moving Average)

Smooths recent observations with exponential decay, making it sensitive to small but consistent changes. The smoothing parameter λ controls the trade-off between responsiveness and false alarm rate.

python

from vfairness.agents import TemporalTracker

tracker = TemporalTracker()

# Feed one turn (or one production window) at a time
for turn, batch in enumerate(daily_agent_logs):
    tracker.record_turn(turn, batch.outcomes_group_a, batch.outcomes_group_b)

# CUSUM: sustained shifts, even small ones
cusum = tracker.detect_drift_cusum(threshold=0.5, drift_limit=5.0)
if cusum["has_drift"]:
    print(f"CUSUM drift first seen at turn {cusum['drift_point']} "
          f"(max statistic {cusum['max_cusum']:.3f})")

# EWMA: smoothed trajectory against widening control limits
ewma = tracker.detect_drift_ewma(span=5, sigma_limit=3.0)
if ewma["has_drift"]:
    print(f"EWMA left the control limits at turns {ewma['drift_points']}")

# The per-turn trajectory behind both detectors
for point in tracker.compute_trajectory():
    print(f"  turn {point.turn_number}: {point.value:.3f} "
          f"(cumulative drift {point.cumulative_drift:.3f})")

Enterprise Features

All agent testing results include production-ready features for audit compliance and system integration.

Audit Trail & Serialization

Every result carries RunMetadata with timestamp, library version, and full parameter snapshot:

python

from vfairness.agents import CorrespondenceTester

tester = CorrespondenceTester(alpha=0.05)
result = tester.analyze_outcomes(outcomes_a, outcomes_b, artifact_type="resume")

# Audit metadata on every result
print(result.metadata.timestamp)        # ISO 8601 UTC
print(result.metadata.library_version)  # the installed vfairness version
print(result.metadata.parameters)       # {"alpha": 0.05, "artifact_type": "resume", ...}

# Serialize for database storage or reporting
data = result.to_dict()    # Plain dict
json_str = result.to_json()  # JSON string

Structured Logging

All operations emit structured logs via Python's logging module:

python

import logging
logging.basicConfig(level=logging.INFO)

# Operations now emit structured logs:
# INFO:vfairness.agents.correspondence:Starting correspondence test with 50 samples per group
# INFO:vfairness.agents.tool_bias:Analyzing 3 tool types across 100 traces
# WARNING:vfairness.agents.pipeline:Sample size (12) below recommended minimum of 25

← Getting Started Multi-Agent Testing →