Overview

This walkthrough demonstrates a complete fairness audit of a machine learning model using the vfairness library. Every phase produces concrete artifacts — code, metrics, and SVG visualizations — that together form the documentation required for EU AI Act compliance (Annex III: creditworthiness scoring).

What you will build

By following this guide, you will produce: a validated dataset audit, bias detection report, fair-transformed features, a calibrated model, full fairness metrics with confidence intervals, robustness tests, CI/CD gate decisions, monitoring dashboards, executive/operational/technical reports, and an EU AI Act–compliant model card — all backed by 24 production-grade SVG artifacts.

The Scenario

Credit Scoring Model — EU AI Act High-Risk System

A financial institution deploys a binary classifier to approve or deny loan applications. Under the EU AI Act (Annex III, Section 5b), creditworthiness scoring is classified as high-risk AI, requiring comprehensive fairness documentation, ongoing monitoring, and human oversight.

50,000
Applications
3
Protected Attributes
24
SVG Artifacts
10
Audit Phases
python 
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier

# ── Load the credit scoring dataset ──
df = pd.read_csv("credit_applications.csv")

# Protected attributes
PROTECTED = ["gender", "race", "age_group"]
OUTCOME   = "approved"
FEATURES  = [c for c in df.columns if c not in PROTECTED + [OUTCOME, "applicant_id"]]

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    df[FEATURES], df[OUTCOME], test_size=0.3, random_state=42, stratify=df[OUTCOME]
)
sensitive_test = df.loc[X_test.index, PROTECTED]

# Baseline model
model = GradientBoostingClassifier(n_estimators=200, max_depth=4, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

Audit Pipeline

The ten phases below map directly to the vfairness module architecture. Each phase produces artifacts that feed into subsequent phases.

1Data
Validation
2Bias
Detection
3Feature
Engineering
4Fair
Training
5Calibration
6Metrics &
Testing
7CI/CD
Gating
8Monitoring
9Dashboards
10Model
Card
1 Preprocessing

Data Validation

Before touching the model, validate the training data for representation balance, missing value patterns, and proxy correlations. The DataBiasValidator runs structural checks and flags issues by severity.

python 
from vfairness.operations.cicd import DataBiasValidator, DataValidationConfig

validator = DataBiasValidator(config=DataValidationConfig(
    min_group_size=50,
    max_missing_rate=0.05,
    proxy_correlation_threshold=0.3,
))
result = validator.validate(df, protected_attrs=PROTECTED, outcome_col=OUTCOME)

print(f"Status: {result.status}")       # PASS, FAIL, or CONDITIONAL
print(f"Issues: {len(result.issues)}")   # List of ValidationIssue objects

# ── Generate SVG artifact ──
from vfairness.rendering import data_validation_to_svg

data_validation_to_svg(
    result,
    explanation="Pre-training data quality assessment for the credit scoring dataset. "
                "Checks group representation, missing patterns, and proxy correlations.",
    save_path="artifacts/01_data_validation.svg",
)
Data Validation Report
Data Validation Report data_validation_to_svg()
Findings

1 critical issue (representation imbalance in age_group), 2 warnings (elevated missing rates in income fields), 2 informational notes. The audit proceeds with documented caveats.

2 Preprocessing

Bias Detection & Auto-Discovery

Run the auto-discovery scanner to confirm that protected attributes are correctly identified, then execute a full bias audit covering historical patterns, representation, statistical disparities, and proxy variable detection.

python 
from vfairness.evaluation.vfairness_metrics import (
    detect_protected_attributes,
    scan_fairness_violations,
    discover_intersectional_groups,
)
from vfairness.preprocessing.bias_detection import BiasDetector
from vfairness.rendering import auto_discovery_to_svg, bias_audit_to_svg

# ── Auto-discovery: confirm protected attributes ──
candidates = detect_protected_attributes(df, outcome_col=OUTCOME)
violations = scan_fairness_violations(y_test, y_pred, sensitive_test["gender"])
intersectional = discover_intersectional_groups(
    df, protected_attrs=PROTECTED, outcome_col=OUTCOME
)

auto_discovery_to_svg(
    candidates=candidates,
    violations=violations,
    group_advantages=intersectional,
    explanation="Automated scan confirming gender, race, and age_group as protected "
                "attributes with confidence scores above 0.70.",
    save_path="artifacts/02a_auto_discovery.svg",
)

# ── Full bias audit ──
detector = BiasDetector(df, protected_attributes=PROTECTED, outcome_column=OUTCOME)
audit = detector.full_audit()

bias_audit_to_svg(
    audit.to_dict(),
    explanation="Comprehensive pre-training bias audit covering historical patterns, "
                "representation balance, statistical disparities, and proxy detection.",
    save_path="artifacts/02b_bias_audit.svg",
)
Auto-Discovery Scanner
Auto-Discovery Scanner auto_discovery_to_svg()
Bias Audit Dashboard
Bias Audit Dashboard bias_audit_to_svg()
Findings

Significant demographic parity violation detected for gender (0.15, threshold 0.10). Intersectional analysis reveals female × minority subgroup is most disadvantaged (positive rate 0.22 vs overall 0.34). Two high-severity violations flagged for immediate attention.

3 Preprocessing

Feature Engineering

Identify proxy variables (features that leak protected attribute information) and apply fairness-aware transformations to reduce correlations while preserving predictive power.

python 
from vfairness.preprocessing.feature_engineering import FeatureEngineeringAnalyzer
from vfairness.rendering import (
    proxy_risk_to_svg,
    correlation_heatmap_to_svg,
    transformation_comparison_to_svg,
)

analyzer = FeatureEngineeringAnalyzer(
    df, protected_attributes=PROTECTED, feature_columns=FEATURES
)
report = analyzer.full_analysis()

# ── Proxy risk assessment ──
proxy_risk_to_svg(
    report.proxy_results,
    explanation="Proxy variables ranked by risk level. ZIP code shows critical "
                "proxy risk (r=0.72 with race), requiring transformation.",
    save_path="artifacts/03a_proxy_risk.svg",
)

# ── Correlation heatmap ──
correlation_heatmap_to_svg(
    report.correlation_matrix,
    explanation="Feature-to-protected-attribute correlation matrix revealing "
                "indirect information leakage pathways.",
    save_path="artifacts/03b_correlation_heatmap.svg",
)

# ── Apply transformation and compare ──
X_fair = analyzer.transform(method="correlation_reduction")

transformation_comparison_to_svg(
    {"before": report.correlation_matrix, "after": X_fair.correlation_matrix},
    explanation="Before-and-after comparison showing correlation reduction "
                "across all features. Average proxy correlation reduced from 0.31 to 0.08.",
    save_path="artifacts/03c_transformation_comparison.svg",
)
Proxy Risk Assessment
Proxy Risk Assessment proxy_risk_to_svg()
Correlation Heatmap
Correlation Heatmap correlation_heatmap_to_svg()
Transformation Comparison
Transformation Comparison transformation_comparison_to_svg()
Findings

Correlation reduction successfully applied. ZIP code proxy risk reduced from CRITICAL to LOW. Average feature-attribute correlation dropped from 0.31 to 0.08, below the 0.10 threshold.

4 Training

Fair Training

Retrain the model using fairness-aware loss functions that penalize demographic parity violations during optimization. Track both task performance and fairness metrics across training epochs.

python 
from vfairness.in_processing import FairnessAwareBCELoss, DemographicParityLoss
from vfairness.rendering import training_report_to_svg, method_comparison_to_svg

# ── Fairness-aware training ──
# (Using PyTorch wrapper around the GBM or a neural net)
loss_fn = FairnessAwareBCELoss(
    fairness_loss=DemographicParityLoss(),
    fairness_weight=0.3,   # Balance task loss vs fairness penalty
)

# ... training loop produces training_history dict ...

training_report_to_svg(
    training_history,
    explanation="Training convergence showing task loss (BCE) and fairness penalty "
                "(demographic parity) over 50 epochs. Fairness weight lambda=0.3.",
    save_path="artifacts/04a_training_report.svg",
)

# ── Compare methods ──
method_comparison_to_svg(
    [baseline_metrics, reweighted_metrics, adversarial_metrics],
    explanation="Three debiasing approaches compared: baseline, sample reweighting, "
                "and adversarial debiasing. Adversarial achieves best fairness-accuracy tradeoff.",
    save_path="artifacts/04b_method_comparison.svg",
)
Training Report
Training Report training_report_to_svg()
Method Comparison
Method Comparison method_comparison_to_svg()
Findings

Fairness-aware training reduces demographic parity difference from 0.15 to 0.06 with only 1.2% accuracy trade-off (AUC: 0.847 → 0.836). The adversarial debiasing method achieves the best Pareto-optimal point.

5 Post-Processing

Calibration

Ensure predicted probabilities are well-calibrated across all demographic groups. Poor calibration means a "70% approval probability" means different things for different groups — a direct fairness violation.

python 
from vfairness.post_processing.calibration import CalibrationAnalyzer
from vfairness.rendering import (
    reliability_diagram_to_svg,
    calibration_report_to_svg,
    pareto_frontier_to_svg,
)

cal = CalibrationAnalyzer(y_test, y_prob, sensitive_test["gender"])
cal_report = cal.full_analysis()

# ── Reliability diagram (overall) ──
reliability_diagram_to_svg(
    y_test, y_prob,
    explanation="Reliability diagram showing predicted vs actual probabilities. "
                "Deviation from the diagonal indicates miscalibration.",
    save_path="artifacts/05a_reliability_diagram.svg",
)

# ── Full calibration report with group analysis ──
calibration_report_to_svg(
    cal_report.to_dict(),
    explanation="Group-wise calibration analysis. ECE computed per demographic group "
                "to detect differential calibration quality.",
    save_path="artifacts/05b_calibration_report.svg",
)

# ── Calibrate and check trade-offs ──
y_calibrated = cal.calibrate(method="isotonic")
tradeoff = cal.analyze_tradeoffs()

pareto_frontier_to_svg(
    tradeoff.pareto_points,
    explanation="Pareto frontier showing the calibration-fairness trade-off space. "
                "Isotonic calibration achieves near-optimal ECE with minimal fairness cost.",
    save_path="artifacts/05c_pareto_frontier.svg",
)
Calibration Report
Calibration Report calibration_report_to_svg()
Reliability Diagram
Reliability Diagram reliability_diagram_to_svg()
Pareto Frontier
Pareto Frontier pareto_frontier_to_svg()
Findings

Isotonic calibration reduces overall ECE from 0.072 to 0.018. Group-wise calibration disparity drops from 0.034 to 0.009 — well below the 0.02 threshold.

6 Evaluation

Comprehensive Metrics & Robustness Testing

This is the core evaluation phase. Compute all fairness metrics with statistical confidence intervals, run permutation tests, sensitivity analysis, and subgroup robustness audits. This phase generates the most artifacts.

python 
from vfairness import FairnessAnalyzer, classification_fairness_report
from vfairness.evaluation.vfairness_metrics import (
    permutation_test, sensitivity_analysis, subgroup_robustness_audit,
    compute_effect_sizes,
)
from vfairness.rendering import (
    radar_chart_to_svg,
    metrics_bar_chart_to_svg,
    confidence_intervals_to_svg,
    effect_sizes_to_svg,
    group_comparison_to_svg,
    robustness_testing_to_svg,
)

# ── Full fairness analysis ──
analyzer = FairnessAnalyzer(
    y_test, y_pred, sensitive_test["gender"],
    y_prob=y_calibrated,
    fair_explainer=True,
)
report = analyzer.get_report(include_ci=True, include_explanations=True)

# ── Radar chart: all metrics at a glance ──
radar_chart_to_svg(
    report,
    explanation="Multi-metric radar chart showing demographic parity, equalized odds, "
                "equal opportunity, predictive parity, and calibration difference.",
    save_path="artifacts/06a_radar_chart.svg",
)

# ── Bar chart: metric values with pass/fail thresholds ──
metrics_bar_chart_to_svg(
    report["metrics"],
    explanation="Bar chart of all fairness metrics against regulatory thresholds. "
                "Green = pass, red = fail.",
    save_path="artifacts/06b_metrics_bar_chart.svg",
)

# ── Confidence intervals ──
confidence_intervals_to_svg(
    report["confidence_intervals"],
    explanation="95% bootstrap confidence intervals for each fairness metric. "
                "If the interval crosses the threshold, the result is inconclusive.",
    save_path="artifacts/06c_confidence_intervals.svg",
)

# ── Effect sizes ──
effects = compute_effect_sizes(y_test, y_pred, sensitive_test["gender"])

effect_sizes_to_svg(
    effects,
    explanation="Cohen's d effect sizes quantifying the practical significance of "
                "group disparities. Values > 0.2 indicate meaningful differences.",
    save_path="artifacts/06d_effect_sizes.svg",
)

# ── Group comparison ──
group_comparison_to_svg(
    report["group_metrics"],
    explanation="Side-by-side comparison of TPR, FPR, precision, and approval rate "
                "across all demographic groups.",
    save_path="artifacts/06e_group_comparison.svg",
)

# ── Robustness testing ──
perm = permutation_test(y_test, y_pred, sensitive_test["gender"])
sens = sensitivity_analysis(y_test, y_pred, sensitive_test["gender"])
audit = subgroup_robustness_audit(y_test, y_pred, sensitive_test["gender"])

robustness_testing_to_svg(
    permutation_results=[perm],
    sensitivity_results=[sens],
    subgroup_results=[audit],
    explanation="Robustness validation: permutation tests confirm statistical "
                "significance, sensitivity analysis shows metric stability under "
                "data perturbation, subgroup audit identifies vulnerable populations.",
    save_path="artifacts/06f_robustness_testing.svg",
)
Fairness Radar Chart
Fairness Radar Chart radar_chart_to_svg()
Metrics Bar Chart
Metrics Bar Chart metrics_bar_chart_to_svg()
Confidence Intervals
Confidence Intervals confidence_intervals_to_svg()
Effect Sizes
Effect Sizes effect_sizes_to_svg()
Group Comparison
Group Comparison group_comparison_to_svg()
Robustness Testing
Robustness Testing Dashboard robustness_testing_to_svg()
Findings

After fair training and calibration: demographic parity difference = 0.06 (pass), equalized odds difference = 0.04 (pass), equal opportunity difference = 0.03 (pass). All permutation tests confirm statistical significance (p < 0.001). Robustness score: 0.72 (MARGINAL → acceptable with documentation).

7 Operations

CI/CD Gating

Integrate fairness checks into your deployment pipeline. The ModelFairnessGate produces a binary PASS/FAIL decision, while the hierarchical gate evaluates intersectional subgroups with per-group thresholds.

python 
from vfairness.operations.cicd import ModelFairnessGate, GateConfig
from vfairness.operations.cicd.gate import evaluate_hierarchical, HierarchicalGateConfig
from vfairness.rendering import cicd_pipeline_to_svg, hierarchical_gate_to_svg

# ── Standard gate ──
gate = ModelFairnessGate(config=GateConfig(
    thresholds={
        "demographic_parity_difference": 0.10,
        "equalized_odds_difference": 0.10,
        "equal_opportunity_difference": 0.08,
    },
    require_min_samples=30,
))
decision = gate.evaluate(y_test, y_pred, sensitive_test["gender"], y_prob=y_calibrated)

cicd_pipeline_to_svg(
    decision.to_dict(),
    explanation="CI/CD fairness gate decision for production deployment. "
                "All three primary metrics pass their respective thresholds.",
    save_path="artifacts/07a_cicd_pipeline.svg",
)

# ── Hierarchical intersectional gate ──
hier_config = HierarchicalGateConfig(
    global_thresholds={"demographic_parity_difference": 0.10},
    per_intersection_thresholds={"female_minority": {"demographic_parity_difference": 0.15}},
    min_subgroup_size=30,
)
hier_decision = evaluate_hierarchical(
    y_test, y_pred, sensitive_test, config=hier_config
)

hierarchical_gate_to_svg(
    hier_decision,
    explanation="Hierarchical gate evaluating global, per-group, and intersectional "
                "fairness. Flags small-sample subgroups with SmallSampleWarning.",
    save_path="artifacts/07b_hierarchical_gate.svg",
)
CI/CD Pipeline Gate
CI/CD Pipeline Gate cicd_pipeline_to_svg()
Hierarchical Gate
Hierarchical Gate hierarchical_gate_to_svg()
Gate Decision: PASS

All global and per-group thresholds satisfied. One intersectional subgroup (non-binary × minority, n=120) triggers a SmallSampleWarning — documented but not blocking.

8 Operations

Production Monitoring

After deployment, continuously monitor fairness metrics for drift. Set up adaptive thresholds, alert prioritization, and temporal trend analysis.

python 
from vfairness.operations.monitoring import FairnessMonitor, FairnessMonitorConfig
from vfairness.operations.monitoring import FairnessDriftDetector
from vfairness.rendering import monitoring_dashboard_to_svg, drift_report_to_svg

# ── Initialize monitor ──
monitor = FairnessMonitor(config=FairnessMonitorConfig(
    metrics=["demographic_parity_difference", "equalized_odds_difference"],
    window_size="1h",
    alert_thresholds={"demographic_parity_difference": 0.12},
))

# Log production predictions (in real deployment, this runs continuously)
monitor.log_predictions(y_prod_true, y_prod_pred, sensitive_prod)

monitoring_dashboard_to_svg(
    monitor,
    explanation="Real-time fairness monitoring dashboard showing metric trends, "
                "alert status, and drift indicators over the past 24 hours.",
    save_path="artifacts/08a_monitoring_dashboard.svg",
)

# ── Drift detection ──
detector = FairnessDriftDetector()
drift = detector.detect_drift(
    y_prod_true, y_prod_pred, sensitive_prod,
    baseline_metrics=report["metrics"],
)

drift_report_to_svg(
    drift,
    explanation="Drift analysis comparing production metrics against baseline. "
                "No significant drift detected in the first monitoring window.",
    save_path="artifacts/08b_drift_report.svg",
)
Monitoring Dashboard
Monitoring Dashboard monitoring_dashboard_to_svg()
Drift Report
Drift Report drift_report_to_svg()
Findings

No significant fairness drift in the initial monitoring window. Demographic parity difference stable at 0.06 ± 0.02. Adaptive thresholds configured for automated alerting.

9 Reporting

Multi-Tier Reporting

Generate three report tiers for different stakeholders: Executive (board-level summary), Operational (team-level detail), and Technical (full statistical evidence). Each tier has its own SVG dashboard.

python 
from vfairness.operations.reporting import ReportGenerator, ReportConfig, MetricsStore
from vfairness.rendering import reporting_dashboard_to_svg

# ── Store metrics ──
store = MetricsStore()
store.record_metric("demographic_parity_difference", 0.06, group="gender")
store.record_metric("equalized_odds_difference", 0.04, group="gender")
store.record_metric("equal_opportunity_difference", 0.03, group="gender")
store.record_metric("calibration_difference", 0.009, group="gender")

generator = ReportGenerator(config=ReportConfig(title="Credit Scoring Fairness Report"))

# ── Executive report ──
exec_report = generator.generate(store, tier="executive", format="markdown")
reporting_dashboard_to_svg(
    exec_report, tier="executive",
    explanation="Board-level summary: overall fairness health score, key risk indicators, "
                "and compliance status for EU AI Act Article 9 requirements.",
    save_path="artifacts/09a_executive_dashboard.svg",
)

# ── Operational report ──
ops_report = generator.generate(store, tier="operational", format="markdown")
reporting_dashboard_to_svg(
    ops_report, tier="operational",
    explanation="Team-level operational dashboard with metric trends, alert history, "
                "and actionable recommendations for the ML engineering team.",
    save_path="artifacts/09b_operational_dashboard.svg",
)

# ── Technical report ──
tech_report = generator.generate(store, tier="technical", format="json")
reporting_dashboard_to_svg(
    tech_report, tier="technical",
    explanation="Full statistical evidence including confidence intervals, effect sizes, "
                "p-values, and robustness test results for regulatory documentation.",
    save_path="artifacts/09c_technical_dashboard.svg",
)
Executive Dashboard
Executive Dashboard reporting_dashboard_to_svg(tier="executive")
Operational Dashboard
Operational Dashboard reporting_dashboard_to_svg(tier="operational")
Technical Dashboard
Technical Dashboard reporting_dashboard_to_svg(tier="technical")
10 Compliance

EU AI Act Model Card

The final deliverable: a FairnessReportCard that consolidates all audit findings into a single, regulation-ready document. This satisfies EU AI Act Articles 9, 11, 13, and 15 requirements for high-risk AI systems.

python 
from vfairness.operations.cicd.gate import FairnessReportCard
from vfairness.rendering import report_card_to_svg

card = FairnessReportCard(
    model_name="CreditScore-GBM-v2.1",
    model_version="2.1.0",
    description="Gradient Boosted Machine for credit application scoring. "
                "Binary classification: approved (1) vs denied (0).",
    intended_use="Automated pre-screening of loan applications for retail banking. "
                 "Human review required for all denial decisions.",
    risk_classification="HIGH",  # EU AI Act Annex III, Section 5b
    protected_attributes=PROTECTED,
    fairness_metrics=report["metrics"],
    training_data_summary={
        "size": 50_000,
        "source": "Internal application database, 2022-2024",
        "preprocessing": "Correlation reduction applied to proxy features",
    },
    evaluation_summary={
        "test_size": 15_000,
        "bootstrap_samples": 1_000,
        "ci_level": 0.95,
    },
    gate_decision=decision.to_dict(),
    monitoring_config={
        "drift_detection": "enabled",
        "alert_thresholds": {"demographic_parity_difference": 0.12},
        "review_cadence": "monthly",
    },
    human_oversight="All denial decisions reviewed by a human credit officer. "
                    "Quarterly bias review board meetings.",
    limitations=[
        "Model trained on historical data that may encode past discrimination.",
        "Small sample sizes for non-binary gender group (n=300).",
        "Geographic coverage limited to EU member states.",
    ],
    references=[
        "EU AI Act, Regulation 2024/1689, Annex III Section 5(b)",
        "EBA Guidelines on loan origination and monitoring (EBA/GL/2020/06)",
    ],
)

report_card_to_svg(
    card,
    explanation="EU AI Act-compliant model card consolidating all audit findings, "
                "risk classification, fairness metrics, monitoring configuration, "
                "human oversight procedures, and known limitations.",
    save_path="artifacts/10_model_card.svg",
)
EU AI Act Model Card
EU AI Act Model Card report_card_to_svg()
EU AI Act Requirements Addressed

This model card, combined with the 24 SVG artifacts generated throughout the audit, addresses the following EU AI Act requirements for high-risk AI systems:

  • Article 9 — Risk management system (Phases 1-3, 6)
  • Article 10 — Data governance (Phases 1-3)
  • Article 11 — Technical documentation (All phases)
  • Article 13 — Transparency and information provision (Phases 9-10)
  • Article 14 — Human oversight (Phase 10: oversight section)
  • Article 15 — Accuracy, robustness, cybersecurity (Phases 5-6)
  • Article 72 — Post-market monitoring (Phase 8)

Complete Artifact Inventory

The table below lists every artifact generated during this audit. Each SVG is a self-contained, print-ready visualization suitable for regulatory submissions.

Phase Artifact Function Module
1Data Validation Reportdata_validation_to_svg()Preprocessing
2Auto-Discovery Scannerauto_discovery_to_svg()Preprocessing
2Bias Audit Dashboardbias_audit_to_svg()Preprocessing
3Proxy Risk Assessmentproxy_risk_to_svg()Preprocessing
3Correlation Heatmapcorrelation_heatmap_to_svg()Preprocessing
3Transformation Comparisontransformation_comparison_to_svg()Preprocessing
4Training Reporttraining_report_to_svg()Training
4Method Comparisonmethod_comparison_to_svg()Training
5Reliability Diagramreliability_diagram_to_svg()Post-Processing
5Calibration Reportcalibration_report_to_svg()Post-Processing
5Pareto Frontierpareto_frontier_to_svg()Post-Processing
6Fairness Radar Chartradar_chart_to_svg()Evaluation
6Metrics Bar Chartmetrics_bar_chart_to_svg()Evaluation
6Confidence Intervalsconfidence_intervals_to_svg()Evaluation
6Effect Sizeseffect_sizes_to_svg()Evaluation
6Group Comparisongroup_comparison_to_svg()Evaluation
6Robustness Testingrobustness_testing_to_svg()Evaluation
7CI/CD Pipeline Gatecicd_pipeline_to_svg()Operations
7Hierarchical Gatehierarchical_gate_to_svg()Operations
8Monitoring Dashboardmonitoring_dashboard_to_svg()Operations
8Drift Reportdrift_report_to_svg()Operations
9Executive Dashboardreporting_dashboard_to_svg()Reporting
9Operational Dashboardreporting_dashboard_to_svg()Reporting
9Technical Dashboardreporting_dashboard_to_svg()Reporting
10EU AI Act Model Cardreport_card_to_svg()Reporting

Workflow Integration

The audit above produces artifacts manually. To automate and enforce these checks in your development pipeline, vfairness provides workflow integration tools that plug directly into MLOps, CI/CD, version control, and testing infrastructure. See the Workflow Integration guide for full documentation.

1. Experiment Tracking

Log every evaluation run’s fairness metrics to MLflow or Weights & Biases so you can compare across model versions.

python 
# Option A: explicit logging after Phase 6
from vfairness import log_fairness_to_mlflow
import mlflow

with mlflow.start_run(run_name="credit-gbm-v2.1"):
    logged = log_fairness_to_mlflow(report, prefix="credit_v2.1")
    print(f"Logged {logged} metrics to MLflow")

# Option B: auto-logging decorator — wraps any evaluation function
from vfairness import auto_log_fairness

@auto_log_fairness(backend="wandb", prefix="credit")
def evaluate_model(model, X_test, y_test, sensitive):
    y_pred = model.predict(X_test)
    return y_pred, y_test, sensitive  # triggers automatic fairness logging

2. Pre-Commit Hooks

Enforce documentation standards before code reaches the repository. The hooks validate that fairness configs contain required fields and that model cards include fairness sections.

yaml 
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/your-org/vfairness
    rev: v0.0.8
    hooks:
      - id: vfairness-check-config    # validates fairness JSON configs
      - id: vfairness-check-model-card # ensures model cards include fairness sections

3. pytest Integration

Embed fairness assertions directly in your test suite. Tests fail if any metric exceeds its threshold — same checks as Phase 6, but automated on every commit.

python 
import pytest
from vfairness import assert_fairness, FairnessTestSuite

# Simple assertion
def test_credit_model_fairness():
    assert_fairness(
        y_true, y_pred, sensitive_test["gender"],
        metrics=["demographic_parity_difference", "equalized_odds_difference"],
        thresholds={"demographic_parity_difference": 0.10, "equalized_odds_difference": 0.15},
    )

# Full test suite with JUnit XML export for CI
suite = FairnessTestSuite(
    protected_attributes=["gender", "race", "age_group"],
    metrics=["demographic_parity_difference"],
    thresholds={"demographic_parity_difference": 0.10},
)
results = suite.test_predictions(y_true, y_pred, sensitive_test)
xml = suite.to_junit_xml()  # attach to CI pipeline artifacts

4. CI/CD Automation & PR Comments

The gate decisions from Phase 7 can be posted directly to GitHub as check results and PR comments — no manual artifact inspection needed.

python 
# Post gate result as a GitHub Check (from Phase 7)
check_payload = gate.create_github_check(decision)
# → POST to /repos/{owner}/{repo}/check-runs

# Generate PR comment with full fairness report card
from vfairness import FairnessReportCard
card = FairnessReportCard(decision, model_name="CreditScore-GBM-v2.1")
comment_payload = card.to_github_comment_payload()
# → POST to /repos/{owner}/{repo}/issues/{pr_number}/comments
yaml 
# .github/workflows/fairness-checks.yml (ready-to-use template)
name: Fairness Gate
on: [pull_request]
jobs:
  fairness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install vfairness
      - run: python -m pytest tests/ -m fairness --junitxml=fairness-results.xml
      - run: python scripts/evaluate_gate.py  # runs gate + posts PR comment
Mapping audit phases to workflow tools

Phase 6 (Metrics) → @auto_log_fairness + assert_fairness()
Phase 7 (CI/CD) → create_github_check() + GitHub Actions YAML
Phase 10 (Model Card) → FairnessReportCard.to_github_comment_payload()
Every commit → Pre-commit hooks validate configs + model cards

EU AI Act Compliance Checklist

Each requirement below is addressed by one or more artifacts from this audit. This checklist can be submitted alongside the model card as part of your conformity assessment documentation.

Art. 9 — Risk Management
Data validation, bias detection, proxy risk analysis, and robustness testing establish a comprehensive risk management system.
Art. 10 — Data Governance
Data validation report documents quality metrics, missing patterns, and representation balance across protected groups.
Art. 11 — Technical Documentation
24 SVG artifacts plus model card provide complete technical documentation of the AI system's design, testing, and validation.
Art. 12 — Record-Keeping
MetricsStore with automated logging ensures all fairness metrics are recorded with timestamps for audit trails.
Art. 13 — Transparency
Multi-tier reporting (executive, operational, technical) provides appropriate transparency for all stakeholder levels.
Art. 14 — Human Oversight
Model card documents human review requirements for denial decisions and quarterly bias review board meetings.
Art. 15 — Accuracy & Robustness
Calibration analysis, permutation tests, sensitivity analysis, and subgroup robustness audit demonstrate system reliability.
Art. 72 — Post-Market Monitoring
Continuous monitoring with drift detection, adaptive thresholds, and automated alerts ensures ongoing compliance.
Next Steps

This walkthrough used demo data for illustration. To apply this process to your own model:

  1. Replace the dataset and model with your own
  2. Adjust thresholds per your organization's risk appetite and regulatory guidance
  3. Configure monitoring for your production environment
  4. Schedule periodic re-audits (recommended: quarterly for high-risk systems)
  5. Submit the model card and artifact bundle as part of your conformity assessment

Explore all 49 SVG templates in the SVG Gallery, or learn about the full library architecture in the Getting Started guide.