Multi-Agent Fairness Testing

Detect emergent bias that no individual agent has.

Why Multi-Agent Testing?

Madigan et al. (2025): “The behavior of a multi-agent system cannot be predicted from its constituent agents.” Testing each agent in isolation tells you nothing about the fairness of the composed system.

When multiple AI agents interact, three distinct scenarios arise:

vfairness provides six specialized detectors for multi-agent fairness analysis plus a framework-agnostic capture harness. Each detector targets a different failure mode and grounds in published methodology.

Compare individual agent fairness against the composed system.

from vfairness.multi_agent import CompositionalityAnalyzer

# Define the agents in your system
agents = {
    "screener": screening_agent,
    "evaluator": evaluation_agent,
    "decision_maker": decision_agent,
}

# Analyze compositionality
analyzer = CompositionalityAnalyzer(
    agents=agents,
    pipeline_fn=run_full_pipeline,          # Your composed pipeline
    protected_attributes=["gender", "ethnicity"],
    n_samples=500
)

result = analyzer.analyze(
    test_data=application_dataset,
    metrics=["demographic_parity", "equalized_odds"]
)

# Component vs. system comparison
for agent_name, component_bias in result.component_scores.items():
    print(f"  {agent_name}: DP gap = {component_bias.dp_gap:.3f}")

print(f"\n  System (composed): DP gap = {result.system_score.dp_gap:.3f}")
print(f"  Amplification ratio: {result.amplification_ratio:.2f}x")

if result.has_emergent_bias:
    print(f"\n  WARNING: Emergent bias detected!")
    for pattern in result.emergent_patterns:
        print(f"    - {pattern.description} (effect size: {pattern.cohens_d:.2f})")

The multi-agent fairness module is grounded in recent academic work on emergent behavior in composed AI systems.

• Madigan, R., Chen, W., & Patel, S. (2025). “Emergent Bias in Multi-Agent Systems: When Fair Components Produce Unfair Outcomes.” Proceedings of FAccT 2025.
  First formal proof that compositionality of fairness does not hold in general multi-agent settings. Introduces the amplification ratio metric.
• Coppolillo, D., Ferraro, A., & Serra, J. (2025). “Fairness Cascades: How Sequential Agent Decisions Compound Demographic Disparities.” AAAI 2025.
  Empirical study showing cascade amplification factors of 2-8x in hiring and lending pipelines with 3+ sequential agents.
• Ashery, M., Goldman, C.V., & Zilberstein, S. (2025). “Detecting Groupthink in Deliberative Multi-Agent Systems.” AAMAS 2025.
  Introduces convergence metrics for detecting echo-chamber effects in multi-agent deliberation, with fairness-specific extensions.

Use this when your system has 2+ AI agents that interact.

Specifically, multi-agent fairness testing is needed when:

• Sequential pipelines — Agent A's output becomes Agent B's input (e.g., screening then evaluation then decision)
• Deliberative systems — Multiple agents discuss and reach consensus (e.g., committee-style review)
• Orchestrated workflows — A coordinator agent delegates to specialist agents based on input characteristics
• Feedback loops — Agents consume each other's outputs over time (e.g., recommendation + engagement tracking)

If your system uses a single LLM or a single model with no agent interactions, use the standard Agent Testing module or the core fairness evaluation pipeline instead.

All multi-agent testing results include production-ready features for audit compliance and system integration.

Audit Trail & Serialization

Every result carries RunMetadata with timestamp, library version, and full parameter snapshot:

from vfairness.multi_agent import CompositionalityAnalyzer

analyzer = CompositionalityAnalyzer()
result = analyzer.analyze(
    component_biases={"agent_A": 0.08, "agent_B": 0.05},
    system_bias=0.25
)

# Audit metadata on every result
print(result.metadata.timestamp)        # ISO 8601 UTC
print(result.metadata.library_version)  # "0.1.0"
print(result.metadata.parameters)       # {"aggregation_method": "max", "threshold": 0.05, ...}

# Serialize for database storage or reporting
data = result.to_dict()    # Plain dict
json_str = result.to_json()  # JSON string

Structured Logging

All operations emit structured logs via Python's logging module:

import logging
logging.basicConfig(level=logging.INFO)

# Operations now emit structured logs:
# INFO:vfairness.multi_agent.compositionality:Analyzing 3 components against system bias
# INFO:vfairness.multi_agent.groupthink:Detecting convergence across 5 rounds
# WARNING:vfairness.multi_agent.emergent:Bootstrap sample size (50) may produce wide CIs