Overview

This walkthrough demonstrates a complete fairness audit of a ranking/recommender system using the vfairness library. Unlike binary classifiers, ranking systems introduce unique fairness challenges: exposure allocation, position bias, and feedback loops where biased rankings generate biased engagement data that reinforces the original bias.

What you will build

By following this guide, you will produce: a validated click-data audit, exposure parity analysis, engagement proxy detection, fair re-ranking evaluation, ranking-specific metrics (NDKL, attention fairness), CI/CD gates with ranking thresholds, feedback-loop monitoring, an A/B experiment comparing fair vs. baseline ranking, causal decomposition of ranking bias, and an EU AI Act–compliant model card — all backed by 24 production-grade SVG artifacts.

The Scenario

RK
Ranking-specific challenges: In a ranking system, fairness is not just about who gets selected but where they appear. A candidate ranked 50th is technically “included” but receives near-zero recruiter attention. Position bias means the top 10 slots capture >80% of clicks. Exposure parity — ensuring each demographic group receives its fair share of attention-weighted positions — is the primary fairness objective. Additionally, click-through data used for model training encodes historical recruiter preferences, creating feedback loops that amplify initial biases over time.

Job Platform Candidate Ranking — EU AI Act High-Risk System

A professional hiring platform deploys a learning-to-rank model (LambdaMART) to order candidates for recruiter search results. Under the EU AI Act (Annex III, Section 4a), AI systems intended to be used for recruitment or selection of natural persons, particularly for screening, filtering, or ranking candidates, are classified as high-risk AI, requiring comprehensive fairness documentation, ongoing monitoring, and human oversight.

120,000
Candidate Profiles
3
Protected Attributes
24
SVG Artifacts
10
Audit Phases
python 
import pandas as pd
import numpy as np
from lightgbm import LGBMRanker

# ── Load the candidate ranking dataset ──
df = pd.read_csv("candidate_profiles.csv")
clicks = pd.read_csv("recruiter_clicks.csv")  # historical engagement data

# Protected attributes
PROTECTED = ["gender", "ethnicity", "age_group"]
FEATURES  = ["skills_match", "experience_years", "education_score",
             "profile_completeness", "past_engagement_rate",
             "recency_score", "keyword_overlap"]

# Merge engagement signals
df = df.merge(clicks.groupby("candidate_id").agg(
    click_rate=("clicked", "mean"),
    impressions=("impression_id", "count"),
), on="candidate_id", how="left").fillna(0)

# Train learning-to-rank model
ranker = LGBMRanker(objective="lambdarank", n_estimators=300, max_depth=6)
ranker.fit(
    X_train[FEATURES], y_train_relevance,
    group=train_query_groups,
    eval_set=[(X_val[FEATURES], y_val_relevance)],
    eval_group=[val_query_groups],
)
scores = ranker.predict(X_test[FEATURES])
rankings = scores.argsort()[::-1]  # descending score → rank

Audit Pipeline

The ten phases below are adapted for ranking systems. Key differences from classification audits: Phase 3 targets engagement-signal proxies, Phase 4 uses post-processing re-ranking (not in-processing), Phase 6 adds ranking-specific metrics (NDKL, exposure parity), and Phase 9 runs a controlled A/B experiment with causal decomposition.

1Data
Validation
2Bias
Detection
3Proxy
Detection
4Fair
Re-ranking
5Score
Calibration
6Ranking
Fairness
7CI/CD
Gating
8Monitoring
9A/B
Testing
10Model
Card
RK
Adapted pipeline: Ranking systems are typically mitigated via post-processing re-ranking rather than in-processing debiasing, because the ranking objective (NDCG, MAP) is non-differentiable and requires specialized constraint optimization. Phase 4 therefore applies fairness-constrained re-ranking after scoring, and Phase 5 calibrates the underlying relevance scores. Phase 9 uses the experimentation framework for A/B comparison of fair vs. baseline ranking with causal analysis.
1 Preprocessing

Data Validation

Validate the training data — both candidate profiles and historical click/engagement data — for representation balance, missing patterns, and position-bias contamination. Click data is particularly vulnerable to feedback loops: candidates historically ranked lower receive fewer impressions, generating less engagement data regardless of their true relevance.

python 
from vfairness.operations.cicd import DataBiasValidator, DataValidationConfig

validator = DataBiasValidator(config=DataValidationConfig(
    min_group_size=200,          # higher bar for ranking (more groups, more queries)
    max_missing_rate=0.03,
    proxy_correlation_threshold=0.25,  # stricter for engagement features
))
result = validator.validate(df, protected_attrs=PROTECTED, outcome_col="click_rate")

print(f"Status: {result.status}")
print(f"Issues: {len(result.issues)}")

# ── Generate SVG artifact ──
from vfairness.rendering import data_validation_to_svg

data_validation_to_svg(
    result,
    explanation="Pre-training data quality assessment for the candidate ranking dataset. "
                "Checks group representation, missing patterns, engagement feature correlations, "
                "and position-bias contamination in historical click data.",
    save_path="artifacts/01_data_validation.svg",
)
Data Validation Report
Data Validation Report data_validation_to_svg()
Findings

1 critical issue (engagement data click_rate is confounded with historical position — candidates from underrepresented groups had 40% fewer average impressions), 2 warnings (low representation for age_group=55+), 2 informational notes. Position-bias decontamination required before model training.

2 Preprocessing

Bias Detection & Auto-Discovery

Run the auto-discovery scanner to identify protected attributes, then perform a full bias audit. For ranking systems, bias detection examines both representation in the candidate pool and exposure distribution in historical rankings — who appears in the top positions and how often.

python 
from vfairness.evaluation.vfairness_metrics import (
    detect_protected_attributes,
    scan_fairness_violations,
    discover_intersectional_groups,
)
from vfairness.preprocessing.bias_detection import BiasDetector
from vfairness.rendering import auto_discovery_to_svg, bias_audit_to_svg

# ── Auto-discovery: confirm protected attributes ──
candidates = detect_protected_attributes(df, outcome_col="click_rate")
violations = scan_fairness_violations(
    y_true=y_test_relevance, y_pred=scores, sensitive=sensitive_test["gender"]
)
intersectional = discover_intersectional_groups(
    df, protected_attrs=PROTECTED, outcome_col="click_rate"
)

auto_discovery_to_svg(
    candidates=candidates, violations=violations,
    group_advantages=intersectional,
    explanation="Automated scan confirming gender, ethnicity, and age_group as protected "
                "attributes. Intersectional analysis reveals compounding disadvantage.",
    save_path="artifacts/02a_auto_discovery.svg",
)

# ── Full bias audit ──
detector = BiasDetector(df, protected_attributes=PROTECTED, outcome_column="click_rate")
audit = detector.full_audit()

bias_audit_to_svg(
    audit.to_dict(),
    explanation="Comprehensive bias audit of historical engagement data. Click rates "
                "confounded with exposure — disparities may reflect position bias, not preference.",
    save_path="artifacts/02b_bias_audit.svg",
)
Auto-Discovery Scanner
Auto-Discovery Scanner auto_discovery_to_svg()
Bias Audit Dashboard
Bias Audit Dashboard bias_audit_to_svg()
Findings

Historical click rates show gender disparity of 0.18 (threshold 0.10). Intersectional analysis reveals female × minority × 45+ subgroup received 62% fewer top-10 placements than the overall average. Critical: this disparity is confounded with position bias and cannot be attributed to candidate quality.

3 Preprocessing

Proxy Detection — Engagement Signals

In ranking systems, engagement features (click-through rate, time-on-profile, response rate) are potent proxy variables. They encode historical recruiter preferences that systematically disadvantage certain groups. Detect and mitigate these proxy correlations before training the ranker.

python 
from vfairness.preprocessing.feature_engineering import FeatureEngineeringAnalyzer
from vfairness.rendering import (
    proxy_risk_to_svg, correlation_heatmap_to_svg,
    transformation_comparison_to_svg,
)

analyzer = FeatureEngineeringAnalyzer(
    df, protected_attributes=PROTECTED, feature_columns=FEATURES + ["click_rate"]
)
report = analyzer.full_analysis()

# ── Proxy risk assessment ──
proxy_risk_to_svg(
    report.proxy_results,
    explanation="Engagement features (click_rate, past_engagement_rate) flagged as "
                "critical proxies — they encode historical position bias and recruiter "
                "preference patterns that correlate with protected attributes (r=0.45 with ethnicity).",
    save_path="artifacts/03a_proxy_risk.svg",
)

# ── Correlation heatmap ──
correlation_heatmap_to_svg(
    report.correlation_matrix,
    explanation="Feature-to-protected-attribute correlations. past_engagement_rate and "
                "click_rate show strong proxy signals. profile_completeness shows moderate risk.",
    save_path="artifacts/03b_correlation_heatmap.svg",
)

# ── Apply transformation ──
X_fair = analyzer.transform(method="correlation_reduction")
transformation_comparison_to_svg(
    {"before": report.correlation_matrix, "after": X_fair.correlation_matrix},
    explanation="Correlation reduction applied to engagement features. Average proxy "
                "correlation reduced from 0.29 to 0.06, below the 0.10 threshold.",
    save_path="artifacts/03c_transformation_comparison.svg",
)
Proxy Risk Assessment
Proxy Risk Assessment proxy_risk_to_svg()
Correlation Heatmap
Correlation Heatmap correlation_heatmap_to_svg()
Transformation Comparison
Transformation Comparison transformation_comparison_to_svg()
Findings

Engagement proxy correlations mitigated. click_rate correlation with ethnicity reduced from 0.45 to 0.04. past_engagement_rate reduced from 0.38 to 0.06. All features now below the 0.10 proxy threshold. The ranker will be retrained on decontaminated features.

4 Post-Processing

Fair Re-ranking

Apply fairness-constrained re-ranking as a post-processing step. The ranker produces relevance scores; the re-ranker interleaves candidates to achieve exposure parity while minimizing NDCG loss. Compare multiple re-ranking strategies (unconstrained, proportional, equal exposure) on the accuracy–fairness Pareto frontier.

RK
Why post-processing: Unlike classification where in-processing (adversarial training, fairness-aware loss) is standard, ranking models optimize non-differentiable metrics (NDCG, MAP) that resist gradient-based fairness constraints. Post-processing re-ranking is the dominant approach in the literature (Singh & Joachims 2018, Zehlike et al. 2022) because it separates relevance estimation from fairness allocation.
python 
from vfairness.post_processing import ThresholdOptimizer
from vfairness.rendering import (
    tradeoff_analysis_to_svg, pareto_frontier_to_svg, method_comparison_to_svg,
)

# ── Compare re-ranking strategies ──
strategies = {
    "Unconstrained": {"exposure_constraint": None},
    "Proportional":  {"exposure_constraint": "proportional"},  # exposure ∝ group size
    "Equal Exposure": {"exposure_constraint": "equal"},         # equal exposure per group
}
results = {}
for name, config in strategies.items():
    optimizer = ThresholdOptimizer(constraint=config["exposure_constraint"])
    results[name] = optimizer.optimize(scores, sensitive_test, rankings)

# ── Pareto frontier: NDCG vs. exposure parity ──
pareto_frontier_to_svg(
    results,
    explanation="Accuracy-fairness Pareto frontier. 'Proportional' re-ranking achieves "
                "exposure parity 0.04 with only 2.1% NDCG degradation — the optimal tradeoff.",
    save_path="artifacts/04a_pareto_frontier.svg",
)

# ── Method comparison ──
method_comparison_to_svg(
    results,
    explanation="Head-to-head comparison of three re-ranking strategies across NDCG@10, "
                "exposure parity, NDKL, and attention fairness.",
    save_path="artifacts/04b_method_comparison.svg",
)
Pareto Frontier
Pareto Frontier pareto_frontier_to_svg()
Method Comparison
Method Comparison method_comparison_to_svg()
Findings

Proportional re-ranking selected. NDCG@10 dropped from 0.847 to 0.826 (2.5% loss) while exposure parity improved from 0.22 to 0.04. Equal-exposure was too aggressive (8.1% NDCG loss). The proportional strategy is deployed as the production ranking policy.

5 Post-Processing

Score Calibration

Calibrate the underlying relevance scores so they are comparable across demographic groups. A score of 0.8 should mean the same thing regardless of the candidate’s gender or ethnicity. This is essential for downstream thresholding and for explaining scores to recruiters.

python 
from vfairness.post_processing import FairnessCalibrator
from vfairness.rendering import (
    reliability_diagram_to_svg, calibration_report_to_svg,
    group_calibration_to_svg,
)

calibrator = FairnessCalibrator(method="platt", groupwise=True)
calibrated_scores = calibrator.fit_transform(scores, sensitive_test)
cal_report = calibrator.report()

reliability_diagram_to_svg(
    cal_report,
    explanation="Reliability diagram showing predicted relevance vs. observed engagement "
                "across decile bins. Post-calibration curves align closely with the diagonal.",
    save_path="artifacts/05a_reliability.svg",
)

calibration_report_to_svg(
    cal_report,
    explanation="Calibration metrics pre- and post-adjustment. ECE reduced from 0.09 to 0.02.",
    save_path="artifacts/05b_calibration.svg",
)

group_calibration_to_svg(
    cal_report,
    explanation="Per-group calibration curves. Before adjustment, female candidates were "
                "systematically under-scored by 0.06 points. Post-calibration gap: 0.008.",
    save_path="artifacts/05c_group_calibration.svg",
)
Reliability Diagram
Reliability Diagram reliability_diagram_to_svg()
Calibration Report
Calibration Report calibration_report_to_svg()
Group Calibration
Group Calibration group_calibration_to_svg()
Findings

Group-wise Platt scaling successfully applied. Expected calibration error (ECE) reduced from 0.09 to 0.02. The systematic under-scoring of female candidates eliminated (gap reduced from 0.06 to 0.008). Calibrated scores are now suitable for recruiter-facing explanations.

6 Evaluation

Ranking Fairness Metrics

Evaluate the re-ranked results using ranking-specific fairness metrics that go beyond standard classification metrics. These measure not just outcome parity but attention allocation — how much recruiter visibility each group receives based on their position in ranked lists.

RK
Ranking vs. classification metrics: Standard metrics (demographic parity, equalized odds) measure binary outcomes. Ranking metrics measure continuous exposure: Exposure parity compares attention-weighted position shares. NDKL (Normalized Discounted KL-divergence) measures divergence from proportional representation at each rank cutoff. Attention fairness weights positions by a logarithmic attention model (position 1 gets weight 1.0, position 10 gets ~0.3).
python 
from vfairness.evaluation.vfairness_metrics import FairnessAnalyzer
from vfairness.evaluation.vfairness_metrics.ranking import (
    get_ranking_fairness_metrics, get_ranking_group_metrics,
)
from vfairness.evaluation.vfairness_metrics.robustness import (
    permutation_test, sensitivity_analysis, subgroup_audit,
)
from vfairness.rendering import (
    ranking_fairness_to_svg, radar_chart_to_svg, metrics_bar_chart_to_svg,
    confidence_intervals_to_svg, effect_sizes_to_svg, group_comparison_to_svg,
    robustness_testing_to_svg, intersectional_analysis_to_svg,
)

# ── Ranking-specific metrics ──
ranking_result = get_ranking_fairness_metrics(
    rankings=fair_rankings, sensitive=sensitive_test["gender"],
    scores=calibrated_scores,
)
group_metrics = get_ranking_group_metrics(
    rankings=fair_rankings, sensitive=sensitive_test["gender"],
    scores=calibrated_scores,
)

ranking_fairness_to_svg(
    ranking_result, group_metrics,
    explanation="Ranking fairness dashboard: exposure parity 0.04 (pass), NDKL 0.08 (pass), "
                "attention fairness 0.06 (pass). All groups receive proportional visibility.",
    save_path="artifacts/06a_ranking_fairness.svg",
)

# ── Standard fairness metrics on binarized top-k ──
analyzer = FairnessAnalyzer(task_type="classification")
report = analyzer.analyze(
    y_true=(y_test_relevance >= 3).astype(int),  # relevant = top category
    y_pred=(fair_rankings <= 10).astype(int),     # predicted top-10
    sensitive=sensitive_test,
)

radar_chart_to_svg(report, explanation="...", save_path="artifacts/06b_radar.svg")
metrics_bar_chart_to_svg(report, explanation="...", save_path="artifacts/06c_bar_chart.svg")
confidence_intervals_to_svg(report, explanation="...", save_path="artifacts/06d_ci.svg")
effect_sizes_to_svg(report, explanation="...", save_path="artifacts/06e_effects.svg")
group_comparison_to_svg(report, explanation="...", save_path="artifacts/06f_group.svg")

# ── Robustness testing ──
perm = permutation_test(y_test_relevance, fair_rankings, sensitive_test["gender"])
sens = sensitivity_analysis(y_test_relevance, fair_rankings, sensitive_test["gender"])
sub  = subgroup_audit(y_test_relevance, fair_rankings, sensitive_test)

robustness_testing_to_svg(
    perm, sens, sub,
    explanation="Robustness: 3/3 permutation tests pass, sensitivity stable under 5% "
                "perturbation, no flagged subgroups.",
    save_path="artifacts/06g_robustness.svg",
)

# ── Intersectional deep-dive ──
intersectional_analysis_to_svg(
    report,
    explanation="Intersectional exposure analysis across gender × ethnicity × age. "
                "No subgroup falls below 80% of proportional exposure threshold.",
    save_path="artifacts/06h_intersectional.svg",
)
Ranking Fairness Dashboard
Ranking Fairness Dashboard ranking_fairness_to_svg()
Fairness Radar Chart
Fairness Radar Chart radar_chart_to_svg()
Metrics Bar Chart
Metrics Bar Chart metrics_bar_chart_to_svg()
Confidence Intervals
Confidence Intervals confidence_intervals_to_svg()
Effect Sizes
Effect Sizes effect_sizes_to_svg()
Robustness Testing
Robustness Testing robustness_testing_to_svg()
Intersectional Analysis
Intersectional Analysis intersectional_analysis_to_svg()
Findings

All ranking fairness metrics pass. Exposure parity 0.04 (threshold 0.10), NDKL 0.08 (threshold 0.15), attention fairness 0.06 (threshold 0.10). Standard metrics on binarized top-10 also pass: demographic parity 0.05, equalized odds 0.07. Robustness tests confirm stability under perturbation. No intersectional subgroup is flagged.

7 Operations

CI/CD Gating

Configure CI/CD gates with ranking-specific thresholds. The hierarchical gate evaluates standard fairness metrics at Level 1, ranking metrics (exposure parity, NDKL) at Level 2, and intersectional checks at Level 3. A model must pass all levels to be deployed.

python 
from vfairness.operations.cicd import FairnessGate, GateConfig
from vfairness.operations.cicd.gate import (
    evaluate_hierarchical, HierarchicalGateConfig,
)
from vfairness.rendering import cicd_pipeline_to_svg, hierarchical_gate_to_svg

# ── Simple gate ──
gate = FairnessGate(config=GateConfig(
    thresholds={
        "demographic_parity_difference": 0.10,
        "exposure_parity_difference": 0.10,
        "ndkl": 0.15,
    },
    block_on_fail=True,
))
decision = gate.evaluate(report["metrics"])

cicd_pipeline_to_svg(
    decision,
    explanation="CI/CD gate with ranking-specific thresholds. Model passes all checks: "
                "demographic parity 0.05, exposure parity 0.04, NDKL 0.08.",
    save_path="artifacts/07a_cicd.svg",
)

# ── Hierarchical gate ──
hier_config = HierarchicalGateConfig(
    level_1_thresholds={"demographic_parity_difference": 0.10},
    level_2_thresholds={"exposure_parity_difference": 0.10, "ndkl": 0.15},
    level_3_thresholds={"attention_fairness": 0.10},
)
hier_decision = evaluate_hierarchical(report["metrics"], hier_config)

hierarchical_gate_to_svg(
    hier_decision,
    explanation="Three-level hierarchical gate: L1 standard fairness → L2 ranking fairness "
                "→ L3 intersectional attention. All levels PASS.",
    save_path="artifacts/07b_hierarchical.svg",
)
CI/CD Pipeline Gate
CI/CD Pipeline Gate cicd_pipeline_to_svg()
Hierarchical Gate
Hierarchical Gate hierarchical_gate_to_svg()
Findings

All three gate levels passed. The model is cleared for production deployment with the proportional re-ranking policy active.

8 Operations

Monitoring & Feedback Loop Detection

Deploy continuous monitoring with special attention to feedback loops. In ranking systems, biased rankings generate biased click data, which retrains the model to produce even more biased rankings. Monitor temporal trends in exposure parity and alert on divergence patterns that indicate loop amplification.

python 
from vfairness.operations.monitoring import MetricsStore, DriftDetector, AlertManager
from vfairness.rendering import (
    monitoring_dashboard_to_svg, drift_report_to_svg,
    alert_timeline_to_svg, temporal_analysis_to_svg,
)

store = MetricsStore(backend="sqlite", path="ranking_fairness_metrics.db")
detector = DriftDetector(method="cusum", threshold=0.02, window=30)
alerter = AlertManager(channels=["slack", "email"], severity_threshold="warning")

# ── Log daily metrics ──
store.log(report["metrics"], tags={"model": "LambdaMART-v3.1", "policy": "proportional"})

# ── Generate monitoring artifacts ──
monitoring_dashboard_to_svg(
    store.get_history(days=90),
    explanation="90-day monitoring dashboard showing exposure parity, NDKL, and "
                "attention fairness trends. No drift detected.",
    save_path="artifacts/08a_monitoring.svg",
)

drift_report_to_svg(
    detector.report(),
    explanation="CUSUM drift detection across ranking fairness metrics. All metrics "
                "within control limits. No feedback-loop amplification detected.",
    save_path="artifacts/08b_drift.svg",
)

alert_timeline_to_svg(
    alerter.get_history(days=90),
    explanation="Alert timeline for the past 90 days. Two informational alerts for "
                "minor exposure shifts, both auto-resolved within 24 hours.",
    save_path="artifacts/08c_alerts.svg",
)

temporal_analysis_to_svg(
    store.get_history(days=90),
    explanation="Temporal analysis of position distribution stability. No feedback-loop "
                "amplification pattern detected — exposure parity remains stable at 0.04 ± 0.01.",
    save_path="artifacts/08d_temporal.svg",
)
Monitoring Dashboard
Monitoring Dashboard monitoring_dashboard_to_svg()
Drift Report
Drift Report drift_report_to_svg()
Alert Timeline
Alert Timeline alert_timeline_to_svg()
Temporal Analysis
Temporal Analysis temporal_analysis_to_svg()
Findings

90-day monitoring shows stable fairness metrics. No feedback-loop amplification detected — exposure parity remains at 0.04 ± 0.01. CUSUM detector shows all metrics within control limits. Two minor alerts were auto-resolved. Monthly model retraining schedule confirmed safe.

9 Experimentation

A/B Testing & Causal Analysis

Run a controlled A/B experiment comparing the baseline (unfair) ranking against the proportional re-ranking policy. Use the experimentation framework to determine statistical significance, estimate the causal effect of the fairness intervention, and decompose direct vs. mediated effects.

RK
Why experimentation matters for ranking: Observational data alone cannot distinguish whether fair re-ranking causes better outcomes or merely correlates with them. An A/B test with random assignment provides causal evidence. The causal decomposition further reveals whether fairness improvements come from direct re-ordering (moving candidates up/down) or indirect effects mediated through changed recruiter behavior (e.g., more diverse shortlists leading to broader hiring).
python 
from vfairness.operations.experimentation import (
    ExperimentRunner, ExperimentConfig, PowerAnalyzer,
    CausalAnalyzer,
)
from vfairness.rendering import (
    experiment_results_to_svg, experiment_recommendation_to_svg,
    power_analysis_to_svg, causal_decomposition_to_svg,
)

# ── Pre-experiment power analysis ──
power = PowerAnalyzer(effect_size=0.05, alpha=0.05, power=0.80)
power_result = power.analyze(n_per_group=5000)

power_analysis_to_svg(
    power_result,
    explanation="Power analysis: n=5,000 per arm sufficient to detect exposure parity "
                "difference of 0.05 with 80% power at α=0.05.",
    save_path="artifacts/09a_power.svg",
)

# ── Run A/B experiment ──
config = ExperimentConfig(
    control="baseline_ranking",
    treatment="proportional_reranking",
    metric="exposure_parity_difference",
    duration_days=14,
    min_sample_size=5000,
)
runner = ExperimentRunner(config)
result = runner.analyze(control_data, treatment_data)

experiment_results_to_svg(
    result,
    explanation="A/B experiment: proportional re-ranking reduces exposure parity from "
                "0.22 to 0.04 (Δ=−0.18, p<0.001) with no significant NDCG loss (Δ=−0.021, p=0.12).",
    save_path="artifacts/09b_experiment.svg",
)

experiment_recommendation_to_svg(
    result,
    explanation="Recommendation: deploy proportional re-ranking. Statistically significant "
                "fairness improvement with non-significant accuracy impact.",
    save_path="artifacts/09c_recommendation.svg",
)

# ── Causal decomposition ──
causal = CausalAnalyzer()
decomposition = causal.decompose(
    treatment="reranking_policy",
    mediator="recruiter_shortlist_diversity",
    outcome="exposure_parity",
    data=experiment_data,
)

causal_decomposition_to_svg(
    decomposition,
    explanation="Causal decomposition: 62% of fairness improvement is direct (re-ordering), "
                "38% is mediated through changed recruiter behavior (more diverse shortlists).",
    save_path="artifacts/09d_causal.svg",
)
Power Analysis
Power Analysis power_analysis_to_svg()
Experiment Results
Experiment Results experiment_results_to_svg()
Experiment Recommendation
Experiment Recommendation experiment_recommendation_to_svg()
Causal Decomposition
Causal Decomposition causal_decomposition_to_svg()
Findings

Experiment confirms causal fairness improvement. Proportional re-ranking reduces exposure parity from 0.22 to 0.04 (p<0.001). NDCG impact is not statistically significant (p=0.12). Causal decomposition: 62% direct effect (re-ordering), 38% mediated through recruiter behavior. Recommendation: deploy treatment arm to 100% of traffic.

10 Reporting

EU AI Act Model Card

The final deliverable: a FairnessReportCard that consolidates all audit findings into a regulation-ready document. For ranking systems under EU AI Act Annex III Section 4(a), the card must address recruitment-specific transparency requirements including the ranking criteria, exposure allocation methodology, and feedback-loop mitigation strategy.

python 
from vfairness.operations.cicd.gate import FairnessReportCard
from vfairness.rendering import report_card_to_svg

card = FairnessReportCard(
    model_name="CandidateRanker-LambdaMART-v3.1",
    model_version="3.1.0",
    description="Learning-to-rank model for candidate search results. Produces relevance "
                "scores with proportional fairness re-ranking applied post-hoc.",
    intended_use="Ranking job candidates for recruiter search queries on the hiring platform. "
                 "Recruiters see ranked lists and make independent contact decisions.",
    risk_classification="HIGH",  # EU AI Act Annex III, Section 4(a)
    protected_attributes=PROTECTED,
    fairness_metrics={
        **report["metrics"],
        "exposure_parity_difference": 0.04,
        "ndkl": 0.08,
        "attention_fairness": 0.06,
    },
    training_data_summary={
        "size": 120_000,
        "source": "Platform candidate profiles and engagement data, 2023-2025",
        "preprocessing": "Engagement proxy decontamination, position-bias correction",
    },
    evaluation_summary={
        "test_size": 36_000,
        "ab_experiment": "14-day, n=10,000 (5k per arm)",
        "causal_analysis": "Baron-Kenny mediation, 62% direct / 38% mediated",
    },
    gate_decision=decision.to_dict(),
    monitoring_config={
        "drift_detection": "CUSUM on exposure_parity, ndkl, attention_fairness",
        "feedback_loop_detection": "enabled — temporal stability monitoring",
        "alert_thresholds": {"exposure_parity_difference": 0.08},
        "review_cadence": "bi-weekly (ranking systems require faster review cycles)",
    },
    human_oversight="Recruiters make all contact decisions independently. Monthly fairness "
                    "review board with data science, legal, and HR representation. "
                    "Quarterly external audit by independent assessor.",
    limitations=[
        "Model trained on historical engagement data that may encode past recruiter preferences.",
        "Feedback-loop mitigation reduces but does not eliminate amplification risk.",
        "Proportional re-ranking optimized for gender; multi-attribute Pareto is approximate.",
        "Small sample sizes for non-binary gender group (n=850) and 55+ age group (n=1,200).",
    ],
    references=[
        "EU AI Act, Regulation 2024/1689, Annex III Section 4(a)",
        "Singh & Joachims (2018) — Fairness of exposure in rankings",
        "Zehlike et al. (2022) — Fair ranking: a critical review",
    ],
)

report_card_to_svg(
    card,
    explanation="EU AI Act-compliant model card for the ranking system. Addresses Annex III "
                "Section 4(a) recruitment requirements including ranking criteria transparency, "
                "exposure allocation methodology, feedback-loop mitigation, and A/B evidence.",
    save_path="artifacts/10_model_card.svg",
)
EU AI Act Model Card
EU AI Act Model Card report_card_to_svg()
EU AI Act Requirements Addressed

This model card, combined with the 24 SVG artifacts generated throughout the audit, addresses the following EU AI Act requirements for high-risk AI systems in recruitment:

  • Article 9 — Risk management system (Phases 1-3: data validation, bias detection, proxy mitigation)
  • Article 10 — Data governance (Phase 1: engagement data decontamination, position-bias correction)
  • Article 11 — Technical documentation (All phases: 24 SVG artifacts + model card)
  • Article 13 — Transparency (Phase 10: ranking criteria, exposure allocation methodology)
  • Article 14 — Human oversight (Recruiters make independent decisions; bi-weekly review board)
  • Article 15 — Accuracy & robustness (Phase 6: robustness testing, Phase 9: A/B causal evidence)
  • Article 26 — Deployer obligations (Feedback-loop monitoring, quarterly external audit)
  • Article 72 — Post-market monitoring (Phase 8: CUSUM drift detection, temporal stability)

Complete Artifact Inventory

The table below lists every artifact generated during this ranking fairness audit. Each SVG is a self-contained, print-ready visualization suitable for regulatory submissions.

Phase Artifact Function Module
1Data Validation Reportdata_validation_to_svg()Preprocessing
2Auto-Discovery Scannerauto_discovery_to_svg()Preprocessing
2Bias Audit Dashboardbias_audit_to_svg()Preprocessing
3Proxy Risk Assessmentproxy_risk_to_svg()Preprocessing
3Correlation Heatmapcorrelation_heatmap_to_svg()Preprocessing
3Transformation Comparisontransformation_comparison_to_svg()Preprocessing
4Pareto Frontierpareto_frontier_to_svg()Post-Processing
4Method Comparisonmethod_comparison_to_svg()Post-Processing
5Reliability Diagramreliability_diagram_to_svg()Post-Processing
5Calibration Reportcalibration_report_to_svg()Post-Processing
5Group Calibrationgroup_calibration_to_svg()Post-Processing
6Ranking Fairness Dashboardranking_fairness_to_svg()Evaluation
6Fairness Radar Chartradar_chart_to_svg()Evaluation
6Metrics Bar Chartmetrics_bar_chart_to_svg()Evaluation
6Confidence Intervalsconfidence_intervals_to_svg()Evaluation
6Effect Sizeseffect_sizes_to_svg()Evaluation
6Robustness Testingrobustness_testing_to_svg()Evaluation
6Intersectional Analysisintersectional_analysis_to_svg()Evaluation
7CI/CD Pipeline Gatecicd_pipeline_to_svg()Operations
7Hierarchical Gatehierarchical_gate_to_svg()Operations
8Monitoring Dashboardmonitoring_dashboard_to_svg()Operations
8Drift Reportdrift_report_to_svg()Operations
8Alert Timelinealert_timeline_to_svg()Operations
8Temporal Analysistemporal_analysis_to_svg()Operations
9Power Analysispower_analysis_to_svg()Experimentation
9Experiment Resultsexperiment_results_to_svg()Experimentation
9Experiment Recommendationexperiment_recommendation_to_svg()Experimentation
9Causal Decompositioncausal_decomposition_to_svg()Experimentation
10EU AI Act Model Cardreport_card_to_svg()Reporting

Workflow Integration

The audit above produces artifacts manually. To automate and enforce these checks in your development pipeline, vfairness provides workflow integration tools that plug directly into MLOps, CI/CD, version control, and testing infrastructure. For ranking systems with feedback loops, automation is especially important to catch drift before biased rankings reinforce themselves. See the Workflow Integration guide for full documentation.

1. Experiment Tracking

Log every ranker evaluation’s fairness metrics to MLflow or Weights & Biases. Track exposure parity, NDKL, and attention fairness across model versions and re-ranking strategies.

python 
# Option A: explicit logging after Phase 6
from vfairness import log_fairness_to_mlflow
import mlflow

with mlflow.start_run(run_name="lambdamart-reranked-v4.2"):
    logged = log_fairness_to_mlflow(report, prefix="ranking_v4.2")
    print(f"Logged {logged} metrics to MLflow")

# Option B: auto-logging decorator — wraps ranking evaluation
from vfairness import auto_log_fairness

@auto_log_fairness(backend="wandb", prefix="ranking")
def evaluate_ranker(rankings, relevance_labels, sensitive):
    return rankings, relevance_labels, sensitive  # triggers fairness logging

2. Pre-Commit Hooks

Enforce documentation standards before code reaches the repository. Catches missing fairness thresholds in config files and incomplete model cards before they are committed.

yaml 
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/your-org/vfairness
    rev: v0.0.8
    hooks:
      - id: vfairness-check-config    # validates fairness JSON configs
      - id: vfairness-check-model-card # ensures model cards include fairness sections

3. pytest Integration

Embed fairness assertions in your test suite. For ranking systems, test that exposure parity and position fairness hold across demographic groups — same checks as Phase 6, but automated on every commit.

python 
import pytest
from vfairness import assert_fairness, FairnessTestSuite

# Test ranking fairness across candidate demographics
def test_ranking_fairness():
    assert_fairness(
        y_true, y_pred, sensitive_test["gender"],
        metrics=["demographic_parity_difference"],
        thresholds={"demographic_parity_difference": 0.10},
    )

# Full test suite covering all protected attributes
suite = FairnessTestSuite(
    protected_attributes=["gender", "ethnicity", "age_group"],
    metrics=["demographic_parity_difference"],
    thresholds={"demographic_parity_difference": 0.10},
)
results = suite.test_predictions(y_true, y_pred, sensitive_test)
xml = suite.to_junit_xml()  # attach to CI pipeline artifacts

4. CI/CD Automation & PR Comments

Gate decisions from Phase 7 can be posted directly to GitHub as check results and PR comments. For ranking systems, this is critical: every model update or re-ranking parameter change must be fairness-validated before affecting real candidates.

python 
# Post gate result as a GitHub Check (from Phase 7)
check_payload = gate.create_github_check(decision)
# → POST to /repos/{owner}/{repo}/check-runs

# Generate PR comment with full fairness report card
from vfairness import FairnessReportCard
card = FairnessReportCard(decision, model_name="JobRank-LambdaMART-v4.2")
comment_payload = card.to_github_comment_payload()
# → POST to /repos/{owner}/{repo}/issues/{pr_number}/comments
yaml 
# .github/workflows/fairness-checks.yml
name: Ranking Fairness Gate
on: [pull_request]
jobs:
  fairness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install vfairness
      - run: python -m pytest tests/ -m fairness --junitxml=fairness-results.xml
      - run: python scripts/evaluate_ranking_gate.py  # runs gate + posts PR comment
Mapping audit phases to workflow tools

Phase 6 (Ranking Fairness) → @auto_log_fairness + assert_fairness()
Phase 7 (CI/CD) → create_github_check() + GitHub Actions YAML
Phase 9 (A/B Testing) → Log experiment results to MLflow for treatment comparison
Phase 10 (Model Card) → FairnessReportCard.to_github_comment_payload()
Every commit → Pre-commit hooks validate configs + model cards

EU AI Act Compliance Checklist

Each requirement below is addressed by one or more artifacts from this audit. This checklist can be submitted alongside the model card as part of your conformity assessment documentation for recruitment AI under Annex III Section 4(a).

Art. 9 — Risk Management
Data validation, bias detection, engagement proxy analysis, and robustness testing establish a comprehensive risk management system for ranking bias.
Art. 10 — Data Governance
Data validation documents engagement data quality, position-bias contamination, and representation balance across protected groups.
Art. 11 — Technical Documentation
29 SVG artifacts plus model card provide complete technical documentation of the ranking system’s design, fairness methodology, A/B evidence, and causal analysis.
Art. 13 — Transparency
Ranking criteria, exposure allocation methodology, and re-ranking policy documented with full code and visualization artifacts.
Art. 14 — Human Oversight
Recruiters make independent contact decisions. Bi-weekly fairness review board with data science, legal, and HR. Quarterly external audit.
Art. 15 — Accuracy & Robustness
Robustness testing, A/B experimentation, and causal analysis provide strong evidence of system reliability and fairness under perturbation.
Art. 26 — Deployer Obligations
Feedback-loop monitoring, temporal stability analysis, and bi-weekly review cycles ensure the deployer maintains ongoing compliance.
Art. 72 — Post-Market Monitoring
Continuous monitoring with CUSUM drift detection, feedback-loop detection, and automated alerts ensures ongoing ranking fairness.
Next Steps

This walkthrough used demo data for illustration. To apply this process to your own ranking system:

  1. Replace the dataset and ranker with your own learning-to-rank model
  2. Identify engagement features that may serve as proxy variables and decontaminate
  3. Evaluate multiple re-ranking strategies on the accuracy–fairness Pareto frontier
  4. Run an A/B experiment to establish causal evidence before full deployment
  5. Configure feedback-loop monitoring with tighter review cycles (bi-weekly recommended)
  6. Submit the model card and artifact bundle as part of your conformity assessment

Explore all 49 SVG templates in the SVG Gallery, or learn about the full library architecture in the Getting Started guide.