validant.ai vfairness

vfairness Documentation

A comprehensive Python library for measuring fairness in machine learning models and detecting bias.

Get Started API Reference

Choose Your Path

Select the documentation view that matches your role

Business User Guide

For AI Product Managers, Compliance Officers, and Business Stakeholders responsible for delivering fair AI products.

View Business Guide

Developer Documentation

For Data Scientists, ML Engineers, and Developers who need complete API reference, code examples, and integration guides.

View API Reference

Why vfairness?

A full-pipeline fairness library — detect, mitigate, calibrate, and monitor

Detect
35+ metrics, proxy variables, intersectional subgroups, statistical validation
Mitigate
12 fair losses, constraint training, adversarial debiasing
Calibrate
5 calibrators, threshold optimization, group-aware tuning
Monitor
Drift detection, CI/CD gates, adaptive alerts, reporting

What's Inside

16 capabilities that cover every stage of the fairness lifecycle

Full ML Pipeline

Eleven modules covering the entire fairness lifecycle: data preprocessing, feature engineering, training-time interventions, post-processing calibration, evaluation, CI/CD operations, LLM testing, agent testing, and multi-agent testing — with production scorers (VADER sentiment, alt-profanity-check toxicity).

Statistical Rigor

Built-in bootstrap & Bayesian confidence intervals, permutation testing, effect sizes (Cohen's d, odds ratio), and multiple testing corrections. Publication-ready statistical validation.

Training-Time Interventions

12 fairness-aware loss functions, adversarial debiasing, counterfactual losses, constraint-based training (Exponentiated Gradient, Grid Search), 5 regularizers, and scikit-learn compatible FairClassifier/FairRegressor wrappers.

Post-Processing & Calibration

5 calibration methods (Platt, Isotonic, Beta, Temperature, Histogram), group-aware calibrators, threshold optimization, prediction reweighting, and impossibility theorem trade-off diagnostics.

FairExplAIner

Human-readable explanations for every metric. Understand what numbers mean, get severity assessments, and receive actionable recommendations — no fairness PhD required.

Regulatory Compliance

43 historical discrimination patterns across US, EU, EU AI Act & Swiss jurisdictions. Automated risk classification, penalty exposure alerts, and compliance detection for Art. 5 prohibited practices & Annex III high-risk systems.

Intersectional Analysis

Detect hidden disparities at group intersections. Auto-discover protected attributes, identify proxy variables via correlation & mutual information, and audit subgroups for fairness gerrymandering.

MLOps & CI/CD

MLflow integration, pytest assertions, training callbacks for Keras/PyTorch/sklearn, DataBiasValidator, ModelFairnessGate, real-time drift detection, adaptive alert thresholds, and prioritized alert routing for production monitoring.

Privacy-Preserving Reporting

Built-in three-tier privacy scheme: k-anonymity suppression for tiny groups, ε-differential privacy noise for medium groups, exact values for large groups. Fairness dashboards that comply with GDPR Art. 25 and the EU AI Act by design — privacy is on by default.

Clean, Unified API

Consistent analyzer pattern across all modules — FairnessAnalyzer, BiasDetector, CalibrationAnalyzer, FairnessTrainingAnalyzer, ThresholdAnalyzer. Classification, regression, and ranking metrics with sensible defaults.

Publication-Ready SVG Reports

49 templated SVG visualizations — fairness dashboards, bias audits, calibration diagrams, drift reports, Pareto frontiers, and more. Beautiful, self-contained vector graphics ready for papers, presentations, and stakeholder reports.

Fairness A/B Testing

Full experiment framework with per-intersection power analysis, sequential testing (SPRT) for early stopping, and Pareto frontier optimization across fairness–accuracy trade-offs.

Auto-Discovery

Automatically detect protected attributes, identify proxy variables, scan for fairness violations, and discover intersectional subgroups — run a full audit without manual configuration.

LLM Fairness Testing

Test LLMs for bias without training data. Counterfactual prompt testing (9 strategies including persona-based), standardized benchmarks (BBQ, BOLD, HolisticBias), the 8-dimensional DecodingTrust suite (Wang et al. 2023, NeurIPS), output analysis, and non-determinism management for foundation models.

Agent Fairness Testing

Test AI agents for bias in tool selection, RAG retrieval, action outcomes, and delegation patterns. Includes correspondence testing and temporal trajectory tracking across agent lifecycles.

Multi-Agent Fairness Testing

Detect emergent bias in multi-agent systems. Seven analyzers: compositionality, groupthink/echo-chamber, emergent amplification, adversarial collusion (Khan et al. 2023), demographic-conditional delegation routing (Bertrand & Mullainathan 2004), turn-by-turn negotiation drift (Bianchi et al. 2024), and a framework-agnostic MultiAgentRunHarness for autogen / crewai / langgraph.

Enterprise & Audit Ready

Structured logging via Python logging, audit trail metadata (timestamp, version, parameters) on every result, to_dict()/to_json() serialization, progress callbacks for batch operations, and configurable thresholds — ready for EU AI Act Article 12 conformity.

Integrates with your ML stack

Quick Start

Get up and running in minutes

bash 
pip install vfairness
python 
from vfairness import FairnessAnalyzer, classification_fairness_report

# Create analyzer with your model predictions
analyzer = FairnessAnalyzer(
    y_true=actual_outcomes,
    y_pred=model_predictions,
    sensitive_attr=demographic_groups,
    fair_explainer=True  # Enable human-readable explanations
)

# Generate comprehensive report with confidence intervals
report = analyzer.get_report(include_ci=True, n_bootstrap=5000)

# Access metrics
print(f"Demographic Parity: {report['metrics']['demographic_parity_difference']:.3f}")
print(f"Equal Opportunity: {report['metrics']['equal_opportunity_difference']:.3f}")

# Get explanations
for metric, explanation in report['explanations']['metrics'].items():
    print(f"\n{metric}: {explanation['severity']}")
    print(f"  {explanation['evaluation']}")
    print(f"  {explanation['recommendation']}")
View Full Tutorial

Library Architecture

Eleven modules following the ML fairness pipeline from data to production

1
Data & Preprocessing
BiasDetector · FeatureEngineeringAnalyzer · Proxy variable detection · Historical pattern analysis
vfairness.preprocessing 2
Training-Time Interventions
Fairness-aware loss functions · Constraint-based training · Adversarial debiasing
vfairness.in_processing 3
Prediction-Time Interventions
GroupCalibrator · CalibrationAnalyzer · Platt, Isotonic, Beta, Temperature scaling · Trade-off analysis
vfairness.post_processing 4
Evaluation & Measurement
FairnessAnalyzer · Classification, regression, ranking metrics · FairExplAIner · Statistical validation · MLOps integration
vfairness.evaluation 5
Monitoring
FairnessMonitor · FairnessDriftDetector · AdaptiveThresholdManager · TemporalFairnessAnalyzer
vfairness.operations.monitoring 6
Reporting & Dashboards
MetricsStore · FairnessDashboard · ReportGenerator · InteractiveDashboard
vfairness.operations.reporting 7
Experimentation
FairnessExperiment · ExperimentAnalysis · Pareto · Causal Inference (DoWhy — identify, mediate, refute, counterfactual, attribute)
vfairness.operations.experimentation 8
Workflow Integration
ModelFairnessGate · HierarchicalGateConfig · FairnessReportCard · Pre-commit hooks · pytest plugin
vfairness.operations.cicd 9
LLM Fairness Testing
9-strategy counterfactual tester · BBQ / BOLD / HolisticBias · DecodingTrust (8 dims) · OutputAnalyzer · non-determinism
vfairness.llm 10
Agent Fairness Testing
Correspondence testing · Tool bias · RAG bias · Delegation patterns · Temporal trajectory tracking
vfairness.agents 11
Multi-Agent Fairness Testing
Compositionality · Groupthink · Emergent amplification · Adversarial collusion · Delegation routing · Negotiation drift · Framework-agnostic harness
vfairness.multi_agent
flowchart LR A["1 · Data &
Preprocessing"] --> B["2 · Training-Time
Interventions"] B --> C["3 · Prediction-Time
Interventions"] C --> D["4 · Evaluation &
Measurement"] D --> E["5 · Monitoring"] D --> F["6 · Reporting &
Dashboards"] D --> G["7 · Experimentation"] D --> H["8 · Workflow
Integration"] D --> I["9 · LLM Fairness
Testing"] I --> J["10 · Agent Fairness
Testing"] J --> K["11 · Multi-Agent
Testing"] style A fill:#fef3c7,stroke:#d97706,stroke-width:2px,color:#92400e style B fill:#e0f2fe,stroke:#0891b2,stroke-width:2px,color:#0c4a6e style C fill:#e0e7ff,stroke:#6366f1,stroke-width:2px,color:#312e81 style D fill:#dbeafe,stroke:#2c5f7c,stroke-width:2px,color:#1e3a5f style E fill:#d1fae5,stroke:#059669,stroke-width:2px,color:#064e3b style F fill:#fce7f3,stroke:#db2777,stroke-width:2px,color:#831843 style G fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#78350f style H fill:#f1f5f9,stroke:#475569,stroke-width:2px,color:#1e293b style I fill:#fef3c7,stroke:#E67E22,stroke-width:2px,color:#7c3a10 style J fill:#fce7f3,stroke:#E74C3C,stroke-width:2px,color:#7f1d1d style K fill:#f3e8ff,stroke:#9B59B6,stroke-width:2px,color:#581c87

How We Compare

vfairness provides features not found in other libraries

Feature vfairness AIF360 Fairlearn Aequitas
Unified Analyzer Class Yes No No No
Ranking Fairness Metrics Yes No No No
Confidence Intervals Built-in No No No
Bayesian (Small Samples) Yes No No No
FairExplAIner Yes No No No
Auto-Discovery Yes No No No
Data & Preprocessing (Bias Auditing) Full Partial No Partial
Prediction-Time Interventions (Calibration) Full No No No
Operations & Monitoring (CI/CD) Built-in No No No
MLflow Integration Native Manual Manual Manual
pytest Assertions Built-in No No No
EU AI Act Compliance 43 Patterns No No No
Multi-Jurisdiction Coverage US/EU/CH US only No US only

Known Limitations

Transparency about what vfairness does and doesn't do

Pre-Release Software Maturity

vfairness is in active development (v0.8). APIs may change before the stable v1.0.0 release. Not yet published on PyPI.

Install: From source via GitHub. Pin your version in requirements.txt.
Benchmarks Not Auto-Generated Data

Population benchmarks for representation analysis must be user-provided. Defaults are US Census 2020 only.

Source: Census Bureau, Eurostat, BFS, or your domain demographics.
Keyword-Based Matching Detection

Historical pattern and protected attribute detection uses column name keyword matching, not NLP or semantic analysis.

Do: Manually review all columns; rename ambiguous ones.
Small Groups Silently Dropped Statistics

Groups below min_group_size (30) are excluded without warning. Minority and intersectional groups are most affected.

Do: Lower threshold for exploration; use Bayesian CI; verify group counts.
Metrics Are Incompatible Theory

Demographic parity, equalized odds, and calibration cannot all be satisfied simultaneously (Impossibility Theorem). Choose your metric before analysis.

See: Impossibility Theorem in Concepts.
Geographic Data US-Only Coverage

HOLC redlining data covers ~40 US cities only. No European, Swiss, or APAC geographic discrimination data is included.

Do: Build custom geographic risk datasets for non-US jurisdictions.

Detailed limitation callouts appear throughout the documentation, marked with the Limitation badge.

Resources

Getting Started Guide

Installation, first fairness analysis, and basic concepts.

Business User Guide

Comprehensive guide for product managers and compliance officers.

API Reference

Complete documentation of all functions, classes, and parameters.

Core Concepts

Deep dives into fairness definitions, metrics, and methodology.

SVG Templates

Interactive gallery of fairness visualisations, dashboards, and report templates.