Workflow Integration

Integrate fairness checks into every stage of the development lifecycle: experiment tracking, version control, CI/CD pipelines, testing, and collaborative review.

The features below ensure fairness is not a one-time audit but a continuous, automated practice.

Experiment Tracking

vfairness provides first-class integration with both MLflow and Weights & Biases. Both functions accept either a FairnessAnalyzer instance or a raw report dictionary and log metrics, confidence intervals, effect sizes, and group statistics.

MLflow

import mlflow
from vfairness import FairnessAnalyzer, log_fairness_to_mlflow

with mlflow.start_run():
    analyzer = FairnessAnalyzer(y_true, y_pred, gender)
    logged = log_fairness_to_mlflow(analyzer, prefix="model_v1")
    print(f"Logged {len(logged)} metrics")

Weights & Biases NEW

import wandb
from vfairness import FairnessAnalyzer, log_fairness_to_wandb

wandb.init(project="fairness-audit")
analyzer = FairnessAnalyzer(y_true, y_pred, gender)
logged = log_fairness_to_wandb(analyzer, prefix="model_v1")
wandb.finish()

Auto-Logging Decorator NEW

from vfairness import auto_log_fairness

@auto_log_fairness(backend="mlflow", prefix="eval")
def evaluate_model(X_test, y_test, sensitive_attr):
    y_pred = model.predict(X_test)
    return y_pred, y_test, sensitive_attr  # (y_pred, y_true, sensitive_attr)

# Fairness metrics are automatically logged when called
evaluate_model(X_test, y_test, gender)

Version Control Hooks

vfairness provides pre-commit hooks that validate fairness-related files before every commit, ensuring documentation and configuration standards are met.

Pre-commit Configuration

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/your-org/vfairness
    rev: v0.0.8
    hooks:
      - id: vfairness-check-config   # Validates fairness config files
      - id: vfairness-check-model-card  # Checks model cards for fairness sections

Hooks

CI/CD Fairness Gates

The ModelFairnessGate class acts as a deployment gate in CI/CD pipelines, evaluating model fairness against configurable thresholds and deciding whether deployment should proceed.

from vfairness import ModelFairnessGate

gate = ModelFairnessGate(
    metrics=['demographic_parity_difference', 'equalized_odds_difference'],
    thresholds={
        'demographic_parity_difference': 0.1,
        'equalized_odds_difference': 0.1,
    },
    blocking_metrics=['demographic_parity_difference'],
)

decision = gate.evaluate(y_true, y_pred, protected_attr)
if decision.approved:
    deploy_model()
else:
    print(decision.blocking_reasons)

The gate returns a GateDecision with status APPROVED, CONDITIONAL, or BLOCKED. Use gate.create_github_check(decision) to generate a GitHub Checks API payload.

Hierarchical Intersectional Gate NEW

The evaluate_hierarchical() method checks fairness at three levels:

1. Overall — across all groups
2. Single-attribute — per individual protected attribute (gender, race, etc.)
3. Intersectional — combinations of attributes (gender × race)

from vfairness import HierarchicalGateConfig

hconfig = HierarchicalGateConfig(
    check_overall=True,
    check_single_attributes=True,
    check_intersections=True,
    intersection_depth=2,
    min_group_size=30,
    default_intersection_threshold_multiplier=1.2,  # 20% relaxed
    per_intersection_thresholds={
        'Female_Black': {'demographic_parity_difference': 0.15},
    },
)

decision = gate.evaluate_hierarchical(
    y_true, y_pred,
    protected_attrs={'gender': gender, 'race': race},
    hierarchical_config=hconfig,
)

for level, result in decision.level_results.items():
    print(f"{level}: {result.status.value}")

for warning in decision.small_sample_warnings:
    print(f"WARNING: {warning.message}")

Small-Sample Warnings

Groups with fewer samples than min_group_size emit SmallSampleWarning objects instead of silently producing unreliable metrics. Each warning includes the group name, sample size, and recommended minimum.

pytest Integration

vfairness provides multiple layers of pytest integration, from simple assertions to a full test suite with JUnit XML export.

assert_fairness()

# test_fairness.py
from vfairness import assert_fairness

def test_model_is_fair():
    assert_fairness(
        y_true, y_pred, gender,
        metrics=['demographic_parity_difference'],
        thresholds={'demographic_parity_difference': 0.1},
    )

FairnessTestSuite

from vfairness import FairnessTestSuite

suite = FairnessTestSuite(
    protected_attributes=['gender'],
    metrics=['demographic_parity_difference'],
    thresholds={'demographic_parity_difference': 0.1},
)
results = suite.test_predictions(y_true, y_pred, gender)
xml = suite.to_junit_xml()  # For CI/CD integration

pytest Plugin NEW

The vfairness pytest plugin registers automatically via the pytest11 entry point. It provides:

Fairness Report Card NEW

The FairnessReportCard generates structured markdown reports for automated PR comments. It supports both simple and hierarchical gate decisions.

from vfairness import FairnessReportCard

card = FairnessReportCard(decision, model_name="loan-model-v3")
print(card.to_markdown())

# For GitHub API
payload = card.to_github_comment_payload()
# POST to /repos/{owner}/{repo}/issues/{pr_number}/comments

DevOps Templates

vfairness ships with ready-to-use templates for common DevOps platforms.

API Summary

Implementation Overview

A summary of the workflow integration capabilities provided by vfairness.