Sample Assessment — End-to-End Fairness Audit

Overview

This walkthrough demonstrates a complete fairness audit of a machine learning model using the vfairness library. Every phase produces concrete artifacts — code, metrics, and SVG visualizations — that together form the documentation required for EU AI Act compliance (Annex III: creditworthiness scoring).

What you will build

By following this guide, you will produce: a validated dataset audit, bias detection report, fair-transformed features, a calibrated model, full fairness metrics with confidence intervals, robustness tests, CI/CD gate decisions, monitoring dashboards, executive/operational/technical reports, and an EU AI Act–compliant model card — all backed by a complete set of production-grade SVG artifacts, itemised in the Complete Artifact Inventory below.

The Scenario

Credit Scoring Model — EU AI Act High-Risk System

A financial institution deploys a binary classifier to approve or deny loan applications. Under the EU AI Act (Annex III, Section 5b), creditworthiness scoring is classified as high-risk AI, requiring comprehensive fairness documentation, ongoing monitoring, and human oversight.

50,000

Applications

Protected Attributes

SVG Artifacts

Audit Phases

python

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier

# ── Load the credit scoring dataset ──
df = pd.read_csv("credit_applications.csv")

# Protected attributes
PROTECTED = ["gender", "race", "age_group"]
OUTCOME   = "approved"
FEATURES  = [c for c in df.columns if c not in PROTECTED + [OUTCOME, "applicant_id"]]

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    df[FEATURES], df[OUTCOME], test_size=0.3, random_state=42, stratify=df[OUTCOME]
)
sensitive_test = df.loc[X_test.index, PROTECTED]

# Baseline model
model = GradientBoostingClassifier(n_estimators=200, max_depth=4, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

Audit Pipeline

The ten phases below map directly to the vfairness module architecture. Each phase produces artifacts that feed into subsequent phases.

1Data
Validation

2Bias
Detection

3Feature
Engineering

4Fair
Training

5Calibration

6Metrics &
Testing

7CI/CD
Gating

8Monitoring

9Dashboards

10Model
Card

1 Preprocessing

Data Validation

Before touching the model, validate the training data for representation balance, missing value patterns, and outcome disparities. The DataBiasValidator runs structural checks and flags issues by severity. (Proxy correlations are covered in Phases 2 and 3.)

python

from vfairness.operations.cicd import DataBiasValidator, DataValidationConfig

validator = DataBiasValidator(
    protected_attributes=PROTECTED,
    config=DataValidationConfig(
        min_group_fraction=0.05,       # each group must be at least 5% of rows
        min_samples_per_group=50,
        missing_value_threshold=0.05,
    ),
)
result = validator.validate(df, outcome_column=OUTCOME)

print(f"Passed: {result.passed}")        # True / False
print(f"Issues: {len(result.issues)}")   # list of ValidationIssue objects

# ── Generate SVG artifact ──
from vfairness.rendering import data_validation_to_svg

data_validation_to_svg(
    result,
    explanation="Pre-training data quality assessment for the credit scoring dataset. "
                "Checks group representation, missing patterns, and outcome disparity.",
    save_path="artifacts/01_data_validation.svg",
)

Data Validation Report data_validation_to_svg()

Findings

1 critical issue (representation imbalance in age_group), 2 warnings (elevated missing rates in income fields), 2 informational notes. The audit proceeds with documented caveats.

2 Preprocessing

Bias Detection & Auto-Discovery

Run the auto-discovery scanner to confirm that protected attributes are correctly identified, then execute a full bias audit covering historical patterns, representation, statistical disparities, and proxy variable detection.

python

from vfairness.evaluation.vfairness_metrics import (
    detect_protected_attributes,
    scan_fairness_violations,
    discover_intersectional_groups,
)
from vfairness.preprocessing.bias_detection import BiasDetector
from vfairness.rendering import auto_discovery_to_svg, bias_audit_to_svg

# ── Auto-discovery: confirm protected attributes ──
candidates = detect_protected_attributes(df)

df_test = df.loc[X_test.index]   # test rows with protected columns attached
violations = scan_fairness_violations(df_test, y_pred, y_true=y_test.to_numpy())
intersectional = discover_intersectional_groups(df_test, PROTECTED, y_pred)

auto_discovery_to_svg(
    candidates=candidates,
    violations=violations,
    group_advantages=intersectional,
    explanation="Automated scan confirming gender, race, and age_group as protected "
                "attributes, with the disparities found for each.",
    save_path="artifacts/02a_auto_discovery.svg",
)

# ── Full bias audit ──
detector = BiasDetector(df, protected_attributes=PROTECTED, outcome_column=OUTCOME)
audit = detector.full_audit()

bias_audit_to_svg(
    audit,
    explanation="Comprehensive pre-training bias audit covering historical patterns, "
                "representation balance, statistical disparities, and proxy detection.",
    save_path="artifacts/02b_bias_audit.svg",
)

Auto-Discovery Scanner auto_discovery_to_svg()

Bias Audit Dashboard bias_audit_to_svg()

Findings

Significant demographic parity violation detected for gender (0.15, threshold 0.10). Intersectional analysis reveals female × minority subgroup is most disadvantaged (positive rate 0.22 vs overall 0.34). Two high-severity violations flagged for immediate attention.

3 Preprocessing

Feature Engineering

Identify proxy variables (features that leak protected attribute information) and apply fairness-aware transformations to reduce correlations while preserving predictive power.

python

from vfairness.preprocessing.feature_engineering import FeatureEngineeringAnalyzer
from vfairness.rendering import (
    proxy_risk_to_svg,
    correlation_heatmap_to_svg,
    transformation_comparison_to_svg,
)

analyzer = FeatureEngineeringAnalyzer(
    df, protected_attributes=PROTECTED,
    target_column=OUTCOME, feature_columns=FEATURES,
)
report = analyzer.full_analysis()

# ── Proxy risk assessment ──
proxy_risk_to_svg(
    report.proxy_variables,
    explanation="Proxy variables ranked by risk level, flagging features whose "
                "correlation with a protected attribute leaks group information.",
    save_path="artifacts/03a_proxy_risk.svg",
)

# ── Correlation heatmap ──
# correlation_heatmap_to_svg expects a feature-keyed view of the matrix
from types import SimpleNamespace

cm = report.correlation_matrix
heatmap_view = SimpleNamespace(
    features=cm.feature_names,
    protected_attributes=cm.protected_attributes,
    correlations={
        f: {a: float(cm.correlations[a][f]) for a in cm.protected_attributes}
        for f in cm.feature_names
    },
)
correlation_heatmap_to_svg(
    heatmap_view,
    explanation="Feature-to-protected-attribute correlation matrix revealing "
                "indirect information leakage pathways.",
    save_path="artifacts/03b_correlation_heatmap.svg",
)

# ── Apply transformation and compare ──
# transform() returns only the transformed feature columns, so rejoin the
# protected attributes before re-analyzing the fair features.
X_fair = analyzer.transform(method="correlation_reduction")
df_fair = pd.concat([X_fair, df[PROTECTED + [OUTCOME]]], axis=1)

after_cm = FeatureEngineeringAnalyzer(
    df_fair, protected_attributes=PROTECTED,
    target_column=OUTCOME, feature_columns=FEATURES,
).full_analysis().correlation_matrix

def max_abs_corr(matrix):
    """Strongest protected-attribute correlation per feature."""
    return {
        f: max(abs(float(matrix.correlations[a][f])) for a in matrix.protected_attributes)
        for f in matrix.feature_names
    }

transformation_comparison_to_svg(
    max_abs_corr(cm), max_abs_corr(after_cm),
    explanation="Before-and-after comparison showing correlation reduction "
                "across all features.",
    save_path="artifacts/03c_transformation_comparison.svg",
)

Proxy Risk Assessment proxy_risk_to_svg()

Correlation Heatmap correlation_heatmap_to_svg()

Transformation Comparison transformation_comparison_to_svg()

Findings

Correlation reduction successfully applied. ZIP code proxy risk reduced from CRITICAL to LOW. Average feature-attribute correlation dropped from 0.31 to 0.08, below the 0.10 threshold.

4 Training

Fair Training

Retrain the model using fairness-aware loss functions that penalize demographic parity violations during optimization. Track both task performance and fairness metrics across training epochs. The loss functions require PyTorch: install with pip install vfairness[training]; on a base install, constructing them raises an ImportError.

python

from vfairness.in_processing import FairnessAwareBCELoss, DemographicParityLoss
from vfairness.rendering import training_report_to_svg, method_comparison_to_svg

# ── Fairness-aware training ──
# (Using PyTorch wrapper around the GBM or a neural net)
loss_fn = FairnessAwareBCELoss(
    fairness_loss=DemographicParityLoss(),
    fairness_weight=0.3,   # Balance task loss vs fairness penalty
)

# ... training loop produces training_history dict ...

training_report_to_svg(
    training_history,
    explanation="Training convergence showing task loss (BCE) and fairness penalty "
                "(demographic parity) over 50 epochs. Fairness weight lambda=0.3.",
    save_path="artifacts/04a_training_report.svg",
)

# ── Compare methods ──
method_comparison_to_svg(
    [baseline_metrics, reweighted_metrics, adversarial_metrics],
    explanation="Three debiasing approaches compared: baseline, sample reweighting, "
                "and adversarial debiasing. Adversarial achieves best fairness-accuracy tradeoff.",
    save_path="artifacts/04b_method_comparison.svg",
)

Training Report training_report_to_svg()

Method Comparison method_comparison_to_svg()

Findings

Fairness-aware training reduces demographic parity difference from 0.15 to 0.06 with only 1.2% accuracy trade-off (AUC: 0.847 → 0.836). The adversarial debiasing method achieves the best Pareto-optimal point.

5 Post-Processing

Calibration

Ensure predicted probabilities are well-calibrated across all demographic groups. Poor calibration means a "70% approval probability" means different things for different groups — a direct fairness violation.

python

from vfairness.post_processing.calibration import CalibrationAnalyzer
from vfairness.rendering import (
    reliability_diagram_to_svg,
    calibration_report_to_svg,
    pareto_frontier_to_svg,
)

cal = CalibrationAnalyzer(y_test, y_prob, sensitive_test["gender"])
cal_report = cal.full_analysis()

# ── Reliability diagram (overall) ──
reliability_diagram_to_svg(
    y_test, y_prob,
    explanation="Reliability diagram showing predicted vs actual probabilities. "
                "Deviation from the diagonal indicates miscalibration.",
    save_path="artifacts/05a_reliability_diagram.svg",
)

# ── Full calibration report with group analysis ──
calibration_report_to_svg(
    cal_report,
    explanation="Group-wise calibration analysis. ECE computed per demographic group "
                "to detect differential calibration quality.",
    save_path="artifacts/05b_calibration_report.svg",
)

# ── Calibrate and check trade-offs ──
y_calibrated = cal.calibrate(method="isotonic")
tradeoff = cal.analyze_tradeoffs()

pareto_frontier_to_svg(
    [p.calibration_error for p in tradeoff.pareto_points],
    [p.fairness_violation for p in tradeoff.pareto_points],
    explanation="Pareto frontier showing the calibration-fairness trade-off space "
                "across decision thresholds.",
    save_path="artifacts/05c_pareto_frontier.svg",
)

Calibration Report calibration_report_to_svg()

Reliability Diagram reliability_diagram_to_svg()

Pareto Frontier pareto_frontier_to_svg()

Findings

Isotonic calibration reduces overall ECE from 0.072 to 0.018. Group-wise calibration disparity drops from 0.034 to 0.009 — well below the 0.02 threshold.

6 Evaluation

Comprehensive Metrics & Robustness Testing

This is the core evaluation phase. Compute all fairness metrics with statistical confidence intervals, run permutation tests, sensitivity analysis, and subgroup robustness audits. This phase generates the most artifacts.

python

from vfairness import FairnessAnalyzer
from vfairness.evaluation.vfairness_metrics import (
    permutation_test, sensitivity_analysis, subgroup_robustness_audit,
)
from vfairness.rendering import (
    radar_chart_to_svg,
    metrics_bar_chart_to_svg,
    confidence_intervals_to_svg,
    effect_sizes_to_svg,
    group_comparison_to_svg,
    robustness_testing_to_svg,
)

# ── Full fairness analysis ──
analyzer = FairnessAnalyzer(
    y_test, y_pred, sensitive_test["gender"],
    y_prob=y_calibrated,
    fair_explainer=True,
)
report = analyzer.get_report(include_ci=True, include_explanations=True)

# ── Radar chart: all metrics at a glance ──
radar_chart_to_svg(
    report,
    explanation="Multi-metric radar chart showing demographic parity, equalized odds, "
                "equal opportunity, predictive parity, and calibration difference.",
    save_path="artifacts/06a_radar_chart.svg",
)

# ── Bar chart: metric values with pass/fail thresholds ──
metrics_bar_chart_to_svg(
    report,
    explanation="Bar chart of all fairness metrics against regulatory thresholds. "
                "Green = pass, red = fail.",
    save_path="artifacts/06b_metrics_bar_chart.svg",
)

# ── Confidence intervals ──
# The forest plot reads a "confidence_intervals" mapping with lower/upper keys,
# built here from the report's metrics_with_ci entries.
confidence_intervals_to_svg(
    {
        "metrics": report["metrics"],
        "confidence_intervals": {
            m: {"lower": ci["lower_bound"], "upper": ci["upper_bound"]}
            for m, ci in report["metrics_with_ci"].items()
        },
    },
    explanation="95% bootstrap confidence intervals for each fairness metric. "
                "If the interval crosses the threshold, the result is inconclusive.",
    save_path="artifacts/06c_confidence_intervals.svg",
)

# ── Effect sizes ──
effect_sizes_to_svg(
    report,
    explanation="Effect sizes quantifying the practical significance of "
                "group disparities.",
    save_path="artifacts/06d_effect_sizes.svg",
)

# ── Group comparison ──
group_comparison_to_svg(
    report,
    explanation="Side-by-side comparison of positive rate "
                "across all demographic groups.",
    save_path="artifacts/06e_group_comparison.svg",
)

# ── Robustness testing ──
# The robustness tests take a metric function of (y_pred, sensitive_attr)
def parity_gap(y_pred_arr, attr):
    rates = [np.mean(y_pred_arr[attr == g]) for g in np.unique(attr)]
    return max(rates) - min(rates)

gender_test = sensitive_test["gender"].to_numpy()
perm = permutation_test(y_pred, gender_test, parity_gap)
sens = sensitivity_analysis(y_pred, gender_test, parity_gap)
audit = subgroup_robustness_audit(y_pred, sensitive_test, y_true=y_test.to_numpy())

robustness_testing_to_svg(
    permutation_results=[perm],
    sensitivity_results=[sens],
    subgroup_audit=audit,
    explanation="Robustness validation: permutation tests confirm statistical "
                "significance, sensitivity analysis shows metric stability under "
                "data perturbation, subgroup audit identifies vulnerable populations.",
    save_path="artifacts/06f_robustness_testing.svg",
)

Fairness Radar Chart radar_chart_to_svg()

Metrics Bar Chart metrics_bar_chart_to_svg()

Confidence Intervals confidence_intervals_to_svg()

Effect Sizes effect_sizes_to_svg()

Group Comparison group_comparison_to_svg()

Robustness Testing Dashboard robustness_testing_to_svg()

Findings

After fair training and calibration: demographic parity difference = 0.06 (pass), equalized odds difference = 0.04 (pass), equal opportunity difference = 0.03 (pass). All permutation tests confirm statistical significance (p < 0.001). Robustness score: 0.72 (MARGINAL → acceptable with documentation).

7 Operations

CI/CD Gating

Integrate fairness checks into your deployment pipeline. The ModelFairnessGate produces a binary PASS/FAIL decision, while the hierarchical gate evaluates intersectional subgroups with per-group thresholds.

python

from vfairness.operations.cicd import ModelFairnessGate, GateConfig
from vfairness.operations.cicd.gate import HierarchicalGateConfig
from vfairness.rendering import cicd_pipeline_to_svg, hierarchical_gate_to_svg

# ── Standard gate ──
gate = ModelFairnessGate(config=GateConfig(
    thresholds={
        "demographic_parity_difference": 0.10,
        "equalized_odds_difference": 0.10,
        "equal_opportunity_difference": 0.08,
    },
))
decision = gate.evaluate(y_test, y_pred, sensitive_test["gender"])

cicd_pipeline_to_svg(
    gate_decision=decision,
    explanation="CI/CD fairness gate decision for production deployment. "
                "All three primary metrics pass their respective thresholds.",
    save_path="artifacts/07a_cicd_pipeline.svg",
)

# ── Hierarchical intersectional gate ──
hier_config = HierarchicalGateConfig(
    check_intersections=True,
    intersection_depth=2,
    min_group_size=30,
    per_intersection_thresholds={"female_minority": {"demographic_parity_difference": 0.15}},
)
hier_decision = gate.evaluate_hierarchical(
    y_test.to_numpy(), y_pred,
    protected_attrs={"gender": sensitive_test["gender"].to_numpy(),
                     "race": sensitive_test["race"].to_numpy()},
    hierarchical_config=hier_config,
)

hierarchical_gate_to_svg(
    hier_decision,
    explanation="Hierarchical gate evaluating global, per-group, and intersectional "
                "fairness. Flags small-sample subgroups with SmallSampleWarning.",
    save_path="artifacts/07b_hierarchical_gate.svg",
)

CI/CD Pipeline Gate cicd_pipeline_to_svg()

Hierarchical Gate hierarchical_gate_to_svg()

Gate Decision: PASS

All global and per-group thresholds satisfied. One intersectional subgroup (non-binary × minority, n=120) triggers a SmallSampleWarning — documented but not blocking.

8 Operations

Production Monitoring

After deployment, continuously monitor fairness metrics for drift. Set up adaptive thresholds, alert prioritization, and temporal trend analysis.

python

from vfairness.operations.monitoring import FairnessMonitor, FairnessMonitorConfig
from vfairness.operations.monitoring import FairnessDriftDetector
from vfairness.rendering import monitoring_dashboard_to_svg, drift_report_to_svg

# ── Initialize monitor ──
monitor = FairnessMonitor(config=FairnessMonitorConfig(
    window_size=1000,          # rows retained in the sliding window, not a duration
    alert_threshold=0.8,       # minimum acceptable disparate impact ratio
    metrics_to_track=["disparate_impact", "demographic_parity", "equalized_odds"],
))

# Batches are DataFrames carrying the prediction, the label, and the
# protected-attribute columns (in real deployment, this runs continuously)
reference_df = pd.DataFrame({
    "prediction": y_pred,
    "label": y_test.to_numpy(),
    "gender": sensitive_test["gender"].to_numpy(),
})
# stand-in for the next production batch: same columns, fresh rows
production_batch_df = reference_df.sample(500, random_state=7)

monitor.set_reference(reference_df)
window = monitor.update_and_check(production_batch_df)

monitoring_dashboard_to_svg(
    window,
    explanation="Fairness monitoring snapshot for the latest production window: "
                "tracked metrics, per-group rates, and alert status.",
    save_path="artifacts/08a_monitoring_dashboard.svg",
)

# ── Drift detection ──
# The detector compares series of a fairness metric over time, in production
# taken from monitor.get_metric_history(); illustrative values shown here
baseline_metric_series = pd.Series([0.06, 0.05, 0.07, 0.06, 0.05, 0.06, 0.07, 0.06] * 8)
production_metric_series = pd.Series(np.linspace(0.06, 0.13, 64))

detector = FairnessDriftDetector()
detector.set_baseline(baseline_metric_series)
drift = detector.check_drift(production_metric_series, metric="demographic_parity")

drift_report_to_svg(
    drift,
    explanation="Drift analysis comparing production metrics against baseline. "
                "No significant drift detected in the first monitoring window.",
    save_path="artifacts/08b_drift_report.svg",
)

Monitoring Dashboard monitoring_dashboard_to_svg()

Drift Report drift_report_to_svg()

Findings

No significant fairness drift in the initial monitoring window. Demographic parity difference stable at 0.06 ± 0.02. Adaptive thresholds configured for automated alerting.

9 Reporting

Multi-Tier Reporting

Generate three report tiers for different stakeholders: Executive (board-level summary), Operational (team-level detail), and Technical (full statistical evidence). Each tier has its own SVG dashboard.

python

from vfairness.operations.reporting import ReportGenerator, ReportConfig, MetricsStore, ReportTier, OutputFormat
from vfairness.rendering import reporting_dashboard_to_svg

# ── Store metrics ──
store = MetricsStore()
store.ingest_window_metrics(window)   # the Phase 8 monitoring window
store.ingest_from_monitor(monitor)    # full monitor history
store.ingest_drift_result(drift)

generator = ReportGenerator(store, config=ReportConfig(custom_title="Credit Scoring Fairness Report"))

# ── Executive report ──
exec_report = generator.generate(tier=ReportTier.EXECUTIVE, output_format=OutputFormat.MARKDOWN)
reporting_dashboard_to_svg(
    exec_report, tier="executive",
    explanation="Board-level summary: overall fairness health score, key risk indicators, "
                "and compliance status for EU AI Act Article 9 requirements.",
    save_path="artifacts/09a_executive_dashboard.svg",
)

# ── Operational report ──
ops_report = generator.generate(tier=ReportTier.OPERATIONAL, output_format=OutputFormat.MARKDOWN)
reporting_dashboard_to_svg(
    ops_report, tier="operational",
    explanation="Team-level operational dashboard with metric trends, alert history, "
                "and actionable recommendations for the ML engineering team.",
    save_path="artifacts/09b_operational_dashboard.svg",
)

# ── Technical report ──
tech_report = generator.generate(tier=ReportTier.TECHNICAL, output_format=OutputFormat.JSON)
reporting_dashboard_to_svg(
    tech_report, tier="technical",
    explanation="Full statistical evidence including confidence intervals, effect sizes, "
                "p-values, and robustness test results for regulatory documentation.",
    save_path="artifacts/09c_technical_dashboard.svg",
)

Executive Dashboard reporting_dashboard_to_svg(tier="executive")

Operational Dashboard reporting_dashboard_to_svg(tier="operational")

Technical Dashboard reporting_dashboard_to_svg(tier="technical")

10 Compliance

EU AI Act Model Card

The final deliverable: a FairnessReportCard that consolidates all audit findings into a single, regulation-ready document. This satisfies EU AI Act Articles 9, 11, 13, and 15 requirements for high-risk AI systems.

python

from vfairness.operations.cicd.gate import FairnessReportCard
from vfairness.rendering import report_card_to_svg

# The card is built from the hierarchical gate decision produced in Phase 7
card = FairnessReportCard(hier_decision, model_name="CreditScore-GBM-v2.1")
markdown = card.to_markdown()
payload = card.to_github_comment_payload()

# The SVG renderer works from the standard gate decision of Phase 7
report_card_to_svg(
    decision=decision,
    model_name="CreditScore-GBM-v2.1",
    explanation="Fairness report card for the credit scoring model: gate status "
                "and the per-metric evaluations behind it.",
    save_path="artifacts/10_model_card.svg",
)

The card carries the measured evidence: the markdown and GitHub payload cover per-level and intersectional results from the hierarchical decision, and the SVG renders the standard gate decision with its per-metric evaluations. The narrative half of an EU AI Act model card is documentation you write around it, not something the library can compute. For this audit that narrative records: intended use (automated pre-screening of retail loan applications, with human review required for every denial); risk classification HIGH under EU AI Act Annex III Section 5(b); the training data summary (internal application database, correlation reduction applied to proxy features) and the evaluation summary (held-out test split, bootstrap confidence intervals at the 95% level); human oversight (denials reviewed by a credit officer, quarterly bias review board); limitations (historical data may encode past discrimination, small sample size for the non-binary gender group, coverage limited to EU member states); and references to Regulation 2024/1689 and the EBA loan origination guidelines.

EU AI Act Model Card report_card_to_svg()

EU AI Act Requirements Addressed

This model card, combined with the SVG artifacts generated throughout the audit, addresses the following EU AI Act requirements for high-risk AI systems:

Article 9 — Risk management system (Phases 1-3, 6)
Article 10 — Data governance (Phases 1-3)
Article 11 — Technical documentation (All phases)
Article 13 — Transparency and information provision (Phases 9-10)
Article 14 — Human oversight (Phase 10: oversight section)
Article 15 — Accuracy, robustness, cybersecurity (Phases 5-6)
Article 72 — Post-market monitoring (Phase 8)

Complete Artifact Inventory

The table below lists every artifact generated during this audit. Each SVG is a self-contained, print-ready visualization suitable for regulatory submissions.

Phase	Artifact	Function	Module
1	Data Validation Report	`data_validation_to_svg()`	Preprocessing
2	Auto-Discovery Scanner	`auto_discovery_to_svg()`	Preprocessing
2	Bias Audit Dashboard	`bias_audit_to_svg()`	Preprocessing
3	Proxy Risk Assessment	`proxy_risk_to_svg()`	Preprocessing
3	Correlation Heatmap	`correlation_heatmap_to_svg()`	Preprocessing
3	Transformation Comparison	`transformation_comparison_to_svg()`	Preprocessing
4	Training Report	`training_report_to_svg()`	Training
4	Method Comparison	`method_comparison_to_svg()`	Training
5	Reliability Diagram	`reliability_diagram_to_svg()`	Post-Processing
5	Calibration Report	`calibration_report_to_svg()`	Post-Processing
5	Pareto Frontier	`pareto_frontier_to_svg()`	Post-Processing
6	Fairness Radar Chart	`radar_chart_to_svg()`	Evaluation
6	Metrics Bar Chart	`metrics_bar_chart_to_svg()`	Evaluation
6	Confidence Intervals	`confidence_intervals_to_svg()`	Evaluation
6	Effect Sizes	`effect_sizes_to_svg()`	Evaluation
6	Group Comparison	`group_comparison_to_svg()`	Evaluation
6	Robustness Testing	`robustness_testing_to_svg()`	Evaluation
7	CI/CD Pipeline Gate	`cicd_pipeline_to_svg()`	Operations
7	Hierarchical Gate	`hierarchical_gate_to_svg()`	Operations
8	Monitoring Dashboard	`monitoring_dashboard_to_svg()`	Operations
8	Drift Report	`drift_report_to_svg()`	Operations
9	Executive Dashboard	`reporting_dashboard_to_svg()`	Reporting
9	Operational Dashboard	`reporting_dashboard_to_svg()`	Reporting
9	Technical Dashboard	`reporting_dashboard_to_svg()`	Reporting
10	EU AI Act Model Card	`report_card_to_svg()`	Reporting

Workflow Integration

The audit above produces artifacts manually. To automate and enforce these checks in your development pipeline, vfairness provides workflow integration tools that plug directly into MLOps, CI/CD, version control, and testing infrastructure. See the Workflow Integration guide for full documentation.

1. Experiment Tracking

Log every evaluation run’s fairness metrics to MLflow or Weights & Biases so you can compare across model versions.

python

# Option A: explicit logging after Phase 6
from vfairness import log_fairness_to_mlflow
import mlflow

with mlflow.start_run(run_name="credit-gbm-v2.1"):
    logged = log_fairness_to_mlflow(report, prefix="credit_v2.1")
    print(f"Logged {logged} metrics to MLflow")

# Option B: auto-logging decorator — wraps any evaluation function
from vfairness import auto_log_fairness

@auto_log_fairness(backend="wandb", prefix="credit")
def evaluate_model(model, X_test, y_test, sensitive):
    y_pred = model.predict(X_test)
    return y_pred, y_test, sensitive  # triggers automatic fairness logging

2. Pre-Commit Hooks

Enforce documentation standards before code reaches the repository. The hooks validate that fairness configs contain required fields and that model cards include fairness sections.

yaml

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/validantai/vfairness
    rev: v0.0.9  # pin to the vfairness release you install
    hooks:
      - id: vfairness-check-config    # validates fairness JSON configs
      - id: vfairness-check-model-card # ensures model cards include fairness sections

3. pytest Integration

Embed fairness assertions directly in your test suite. Tests fail if any metric exceeds its threshold — same checks as Phase 6, but automated on every commit. By default test_predictions raises on the first failure (pytest style); for CI reporting, pass raise_on_failure=False so the failures land in the JUnit XML and the pipeline gates on that.

python

import pytest
from vfairness import assert_fairness, FairnessTestSuite

# Simple assertion
def test_credit_model_fairness():
    assert_fairness(
        y_true, y_pred, sensitive_test["gender"],
        metrics=["demographic_parity_difference", "equalized_odds_difference"],
        thresholds={"demographic_parity_difference": 0.10, "equalized_odds_difference": 0.15},
    )

# Full test suite with JUnit XML export for CI
suite = FairnessTestSuite(
    protected_attributes=["gender", "race", "age_group"],
    metrics=["demographic_parity_difference"],
    thresholds={"demographic_parity_difference": 0.10},
)
results = suite.test_predictions(
    y_true, y_pred, sensitive_test["gender"],
    attr_name="gender",
    raise_on_failure=False,   # record failures in the XML instead of raising
)
xml = suite.to_junit_xml()  # attach to CI pipeline artifacts

4. CI/CD Automation & PR Comments

The gate decisions from Phase 7 can be posted directly to GitHub as check results and PR comments — no manual artifact inspection needed.

python

# Post gate result as a GitHub Check (from Phase 7)
check_payload = gate.create_github_check(decision)
# → POST to /repos/{owner}/{repo}/check-runs

# Generate PR comment with full fairness report card
from vfairness import FairnessReportCard
card = FairnessReportCard(decision, model_name="CreditScore-GBM-v2.1")
comment_payload = card.to_github_comment_payload()
# → POST to /repos/{owner}/{repo}/issues/{pr_number}/comments

yaml

# .github/workflows/fairness-checks.yml (ready-to-use template)
name: Fairness Gate
on: [pull_request]
jobs:
  fairness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install vfairness
      - run: python -m pytest tests/ -m fairness --junitxml=fairness-results.xml
      - run: python scripts/evaluate_gate.py  # runs gate + posts PR comment

Mapping audit phases to workflow tools

Phase 6 (Metrics) → @auto_log_fairness + assert_fairness()
Phase 7 (CI/CD) → create_github_check() + GitHub Actions YAML
Phase 10 (Model Card) → FairnessReportCard.to_github_comment_payload()
Every commit → Pre-commit hooks validate configs + model cards

EU AI Act Compliance Checklist

Each requirement below is addressed by one or more artifacts from this audit. This checklist can be submitted alongside the model card as part of your conformity assessment documentation.

Art. 9 — Risk Management

Data validation, bias detection, proxy risk analysis, and robustness testing establish a comprehensive risk management system.

Art. 10 — Data Governance

Data validation report documents quality metrics, missing patterns, and representation balance across protected groups.

Art. 11 — Technical Documentation

The SVG artifacts plus the model card provide complete technical documentation of the AI system's design, testing, and validation.

Art. 12 — Record-Keeping

MetricsStore with automated logging ensures all fairness metrics are recorded with timestamps for audit trails.

Art. 13 — Transparency

Multi-tier reporting (executive, operational, technical) provides appropriate transparency for all stakeholder levels.

Art. 14 — Human Oversight

Model card documents human review requirements for denial decisions and quarterly bias review board meetings.

Art. 15 — Accuracy & Robustness

Calibration analysis, permutation tests, sensitivity analysis, and subgroup robustness audit demonstrate system reliability.

Art. 72 — Post-Market Monitoring

Continuous monitoring with drift detection, adaptive thresholds, and automated alerts ensures ongoing compliance.

Next Steps

This walkthrough used demo data for illustration. To apply this process to your own model:

Replace the dataset and model with your own
Adjust thresholds per your organization's risk appetite and regulatory guidance
Configure monitoring for your production environment
Schedule periodic re-audits (recommended: quarterly for high-risk systems)
Submit the model card and artifact bundle as part of your conformity assessment

Explore every SVG template in the SVG Gallery, or learn about the full library architecture in the Getting Started guide.