Multi-Agent Fairness Testing

Detect emergent bias that no individual agent has.

Why Multi-Agent Testing?

Madigan et al. (2025, arXiv:2512.16433): “multi-agent decision systems must be evaluated as holistic entities rather than through reductionist analyses of their constituent components.” Testing each agent in isolation does not establish the fairness of the composed system.

When multiple AI agents interact, three distinct scenarios arise:

vfairness provides six specialized detectors for multi-agent fairness analysis plus a framework-agnostic capture harness. Each detector targets a different failure mode and grounds in published methodology.

Compare individual agent fairness against the composed system.

from vfairness.multi_agent import CompositionalityAnalyzer, EmergentBiasDetector

# Measure each component on its own, then measure the composed system,
# and hand both to the analyzer as plain bias scores.
analyzer = CompositionalityAnalyzer()
result = analyzer.analyze(
    component_biases={"screener": 0.05, "evaluator": 0.03, "decision_maker": 0.04},
    system_bias=0.15,
)

# Component vs. system comparison
for agent_name, component_bias in result.component_scores.items():
    print(f"  {agent_name}: bias = {component_bias:.3f}")

print(f"  System (composed): bias = {result.system_score:.3f}")
print(result.scenario)     # 'amplification' | 'reduction' | 'novel_emergence' | 'consistent'
print(result.divergence)   # system bias minus the aggregated component bias

# For the amplification factor, work from the raw outputs instead
detector = EmergentBiasDetector()
emergent = detector.analyze(component_outputs, system_outputs, groups)
print(f"  Amplification factor: {emergent.amplification_factor:.2f}x")
print(f"  Emergent bias: {emergent.is_emergent}")

The multi-agent fairness module is grounded in recent academic work on emergent behavior in composed AI systems.

• Madigan, M., Kamalaruban, P., Moynihan, G., Kempton, T., Sutton, D., & Burrell, S. (2025). “Emergent Bias and Fairness in Multi-Agent Decision Systems.” arXiv:2512.16433 (preprint).
  Large-scale simulations of financial multi-agent systems showing patterns of emergent bias that cannot be traced to individual agent components; argues these systems must be evaluated as holistic entities.
• Coppolillo, E., Manco, G., & Aiello, L. M. (2025). “Unmasking Conversational Bias in AI Multiagent Systems.” arXiv:2501.14844.
  Echo-chamber effects in multi-agent LLM conversations: interacting agents can converge on biased outcomes that single-model evaluations miss.
• Ashery, A. F., Aiello, L. M., & Baronchelli, A. (2025). “Emergent social conventions and collective bias in LLM populations.” arXiv:2410.08948.
  Collective bias can emerge in a population of interacting LLM agents even when individual agents show no prior bias.

Use this when your system has 2+ AI agents that interact.

Specifically, multi-agent fairness testing is needed when:

• Sequential pipelines — Agent A's output becomes Agent B's input (e.g., screening then evaluation then decision)
• Deliberative systems — Multiple agents discuss and reach consensus (e.g., committee-style review)
• Orchestrated workflows — A coordinator agent delegates to specialist agents based on input characteristics
• Feedback loops — Agents consume each other's outputs over time (e.g., recommendation + engagement tracking)

If your system uses a single LLM or a single model with no agent interactions, use the standard Agent Testing module or the core fairness evaluation pipeline instead.

All multi-agent testing results include production-ready features for audit compliance and system integration.

Audit Trail & Serialization

Every result carries RunMetadata with timestamp, library version, and full parameter snapshot:

from vfairness.multi_agent import CompositionalityAnalyzer

analyzer = CompositionalityAnalyzer()
result = analyzer.analyze(
    component_biases={"agent_A": 0.08, "agent_B": 0.05},
    system_bias=0.25
)

# Audit metadata on every result
print(result.metadata.timestamp)        # ISO 8601 UTC
print(result.metadata.library_version)  # the installed vfairness version
print(result.metadata.parameters)       # {"aggregation_method": "max", "threshold": 0.05, ...}

# Serialize for database storage or reporting
data = result.to_dict()    # Plain dict
json_str = result.to_json()  # JSON string

Structured Logging

All operations emit structured logs via Python's logging module:

import logging
logging.basicConfig(level=logging.INFO)

# Operations now emit structured logs:
# INFO:vfairness.multi_agent.compositionality:Analyzing 3 components against system bias
# INFO:vfairness.multi_agent.groupthink:Detecting convergence across 5 rounds
# WARNING:vfairness.multi_agent.emergent:Bootstrap sample size (50) may produce wide CIs