All notable changes to the Fairness Pipeline Development Toolkit are documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
execute_workflow training controls (runtime-only, not in YAML or CLI):
class_weight (str | dict | None, default "balanced") — passed to baseline and
reductions-path LogisticRegression to avoid degenerate majority-class predictions on
imbalanced data.decision_threshold (float | None, default None) — probability cutoff for binary
predictions via predict_proba; simulates selective classifiers (e.g. hiring screeners).
When None, uses predict() (sklearn default 0.5).StandardScaler in the integrated workflow: fitted once on baseline training features
in run_baseline_measurement(); the same scaler is reused with transform only (never
refit) in run_transform_and_train() so before/after fairness comparisons share identical
scaling.class_weight: Workflow baseline and default reductions base estimator now use
class_weight="balanced" instead of sklearn’s implicit None.docs/api.md, docs/integration_guide.md, docs/playbook-part-five-fairpipe.md, and
DOCS.md document class_weight, decision_threshold, and scaler behavior.pyproject.toml, fairness_pipeline_dev_toolkit.__version__,
docs/conf.py, and docs/VERSIONING.md.execute_workflow random_state: Optional parameter (default 42) to control the stratified
train/test split and downstream model RNGs for reproducible end-to-end runs.fairpipe run-pipeline --random-state (default 42).random_state form field on POST /workflow (default 42).execute_workflow now builds one stratified train/test partition
(WorkflowSplit) shared by baseline measurement, transform-and-train, and final validation,
instead of three separate train_test_split calls. Default random_state=42 preserves prior
behavior for existing callers.LogisticRegression base estimators for the reductions method and
PyTorch training paths (regularized, lagrangian) receive numpy / torch seeds from
random_state.make_latest input from softprops/action-gh-release@v1
so the “Create GitHub Release” step completes without errors.tests/cli/test_run_pipeline.py mock Args objects include random_state for
cmd_run_pipeline compatibility.docs/api.md and docs/playbook-part-five-fairpipe.md document random_state and reproducibility.docs/conf.py and docs/VERSIONING.md set to 0.9.0.features: Optional explicit feature list in YAML / PipelineConfig; when
omitted, execute_workflow auto-selects numeric columns (excluding target and sensitive) with a
logged warning.fairpipe validate: --threshold and --metric for pass/fail (exit 0 / 1) and a
Threshold verdict section in the Markdown report; --metric is required when --threshold
is set. --quiet and non-TTY stdout suppress INFO logs for clean piping.PipelineResult: Structured return from apply_pipeline (data, metadata, sample_weight,
transformers_applied).apply_pipeline return type: Uses PipelineResult instead of a plain tuple. Tuple unpacking
still works but emits DeprecationWarning; prefer attribute access.setup_logging(..., console_stream=...)).execute_workflow: Mixed-type DataFrames no longer break baseline / training feature matrices;
mitigation sample weights from reweighting transformers are applied to training (including
Fairlearn reductions via an internal mixer); sensitive columns are included in the frame passed to
the pipeline when needed for InstanceReweighting.Replace Xt, meta = apply_pipeline(pipe, df) with result = apply_pipeline(pipe, df) then
result.data, result.metadata, and result.sample_weight as needed.
docs/.__version__ sync: fairness_pipeline_dev_toolkit.__version__ and the namespace test now
match pyproject.toml (they had drifted behind earlier releases).DOCS.md, docs/api.md, and docs/integration_guide.md updated to
0.7.4 where they showed stale values.fairpipe.stats shims: fairpipe.stats, fairpipe.stats.bootstrap, fairpipe.stats.bayesian,
fairpipe.stats.effect_size, and fairpipe.stats.multipletests are now importable via the
fairpipe.* compatibility layer, matching the existing shim pattern for other modules.fairpipe.api shims: fairpipe.api, fairpipe.api.app, fairpipe.api.store,
fairpipe.api.models, and fairpipe.api.routes are now importable via fairpipe.*.fairpipe.integration.reporting shim: from fairpipe.integration.reporting import
generate_training_fairness_report (and other reporting helpers) now resolves correctly.log_fairness_metrics export: Added to fairness_pipeline_dev_toolkit.integration.__all__
and the fairpipe.integration shim; previously existed in mlflow_logger but was not exported.assert_fairness comparators: ">=" and ">" were listed in the docstring but not
implemented — any comparator other than "<=" silently applied <. All four comparators
(<=, <, >=, >) are now correctly dispatched; invalid values raise ValueError.fairness_pipeline_dev_toolkit.training: All six exports (ReductionsWrapper,
FairnessRegularizerLoss, LagrangianFairnessTrainer, GroupFairnessCalibrator, sweep_pareto,
plot_pareto) are now resolved via __getattr__, so importing the module no longer fails when
PyTorch or Fairlearn are not installed.FairnessReportingDashboard in fairness_pipeline_dev_toolkit.monitoring:
deferred via __getattr__ so the monitoring module loads cleanly without fairpipe[monitoring].benjamini_hochberg docstring: Clarified that the first return value is adjusted p-values
in ascending-sorted order, not mapped back to the original input order.README.md, DOCS.md, docs/api.md,
and docs/integration_guide.md now import from the fairpipe.* namespace. The
fairness_pipeline_dev_toolkit.* namespace continues to work for backward compatibility.fairpipe.* namespace (e.g. from fairpipe.metrics import FairnessAnalyzer) instead of the
internal fairness_pipeline_dev_toolkit.* namespace.test_namespace.py: Updated hardcoded version string from 0.6.5 →
0.7.1 so the test suite passes against the current release.fairpipe[api]): Optional FastAPI REST server exposing five HTTP endpoints —
GET /health, POST /validate, POST /pipeline, POST /workflow, and GET /results/{run_id}.
Swagger UI available at /docs, ReDoc at /redoc.fairpipe serve CLI command: Start the REST API server with --host, --port, --reload,
and --workers options. Lazy-imports uvicorn and fastapi so the core CLI remains functional
when the api extra is not installed.ResultStore: Thread-safe in-memory LRU result cache (500-entry cap) used by all API routes
to persist results retrievable via GET /results/{run_id}.docker-compose.yml: Run the REST API in Docker with
docker build -t fairpipe-api . && docker run -p 8000:8000 fairpipe-api or docker compose up.tests/api/test_api.py covering all endpoints, thread safety, error
handling, and the “passed=false is not 500” contract. Tests auto-skip when the api extra is
absent. Total test count: 738.ci.yml) now installs .[dev,api] so all API tests run on every push.run_baseline_measurement(). The function accepts optional train_size and random_state and, when config.training and config.fairness_metric are set, trains an unconstrained LogisticRegression and returns baseline_metrics in its result dict. execute_workflow() uses this return value and no longer duplicates baseline computation.run-pipeline output: Enhanced workflow results with explicit pass/fail (PASSED/FAILED with symbol), improvement percentage and “(negative = reduction in unfairness)”, a tabular baseline vs final vs change comparison, and short “Baseline (Step 1)” and “Final (Step 3)” summaries.Unsatisfiable in test_eo_difference_bounds by adding an eo_valid_arrays strategy that generates aligned (y_true, y_pred, sensitive) with at least two groups having both 0 and 1 in y_true by construction, so the test no longer relies on strict assume() filters.fairness-pipeline-dev-toolkit to fairpipe for a shorter, more user-friendly installation command.
pip install fairpipe instead of pip install fairness-pipeline-dev-toolkitfrom fairness_pipeline_dev_toolkit import ...fairpipeThis release simplifies the installation experience by providing a shorter package name while maintaining full backward compatibility for Python code. The package name change only affects the pip install command; all Python imports and CLI usage remain unchanged.
Migration Notes:
pip uninstall fairness-pipeline-dev-toolkitpip install fairpipefrom fairness_pipeline_dev_toolkit.metrics import FairnessAnalyzerfrom fairness_pipeline_dev_toolkit.pipeline.config import PipelineConfigfairpipe commands as beforepip install fairpipe[training]pip install fairpipe[monitoring]pip install fairpipe[adapters]pip install fairpipe[training,monitoring,adapters]mae_parity_difference method to FairlearnAdapter and AequitasAdapter
mae_parity_difference was only implemented in NativeAdapter, causing failures when users selected fairlearn or aequitas backends for regression metricsNativeAdapterThis patch release fixes a bug where regression metrics (mae_parity_difference) would fail when using fairlearn or aequitas backends. The method was expected by FairnessAnalyzer but was missing from these adapters.
Migration Notes:
mae_parity_difference with any backend (native, fairlearn, or aequitas)tests/property_based/test_property_based.py)tests/integration/test_integration_expanded.py) with 20+ edge case scenarioshypothesis>=6.100 to development dependenciesdocs/conf.py, docs/index.rst).github/workflows/docs.yml)readthedocs.yml) as alternative hosting optiondocs/getting_started.md).github/dependabot.yml) for automated dependency updates.github/SECURITY_REVIEW_PROCESS.md).github/workflows/security-review.yml).github/workflows/release.yml)continue-on-error to PyPI publish step to ensure releases are created even if publishing failsThis release focuses on testing infrastructure, documentation automation, and security automation. The enhanced testing ensures robustness across edge cases, the documentation site provides better user experience, and security automation ensures ongoing dependency security.
Migration Notes:
hypothesis package (included in dev dependencies)docs/requirements.txt)TESTPYPI_API_TOKEN secret in GitHub for TestPyPI publishingfonttools>=4.60.2 (CVE-2025-66034)starlette>=0.49.1 (CVE-2025-62727)werkzeug>=3.1.5 (CVE-2025-66221, CVE-2026-21860)virtualenv>=20.36.2 (CVE-2026-22702).github/workflows/security.yml) with weekly scheduled scansSECURITY.md) with vulnerability reporting guidelinesSECURITY_SCAN_RESULTS.md to track remediation statustests/performance/test_performance_suite.py) with baseline performance testsscripts/profile_performance.py) using cProfile for bottleneck identificationfairness_pipeline_dev_toolkit/utils/logging.py) with JSON format supportFAIRPIPE_LOG_LEVEL, FAIRPIPE_LOG_FILE, FAIRPIPE_JSON_LOGS)docs/FEEDBACK.md) with feedback form and guidelines.github/FEEDBACK_REVIEW_PROCESS.md) with priority guidelines and response timeframespyproject.toml and requirements.in to pin security-critical indirect dependenciesdocs/PERFORMANCE.md with information about new performance test suite and profiling toolsThis release focuses on production readiness improvements including security hardening, performance monitoring, observability through structured logging, and community engagement through feedback infrastructure. These enhancements support long-term maintainability and user satisfaction.
Migration Notes:
pytest tests/performance/Performance Documentation: Created comprehensive docs/PERFORMANCE.md with performance benchmarks, optimization tips, scalability considerations, memory usage guidelines, and CI/CD integration examples.
Automated Release Workflow: Added .github/workflows/release.yml for automated PyPI publication and GitHub release creation when version tags are pushed.
Performance Benchmarking in CI: Enhanced CI workflow to run performance benchmarks on Ubuntu, tracking performance regressions and uploading benchmark results as artifacts.
FairnessToolkitError base class with message, context, and suggestion attributesDataValidationError for data validation failuresDependencyError for missing optional dependenciesorchestrator.pyReductionsWrapper, LagrangianFairnessTrainer, FairnessRegularizerLoss) are now only imported when needed[training] or [adapters] extrasAPI Documentation: Updated docs/api.md with complete exception hierarchy documentation, including all new exception types with usage examples.
TrainingError exception type instead of generic ValueError.This release significantly improves the developer experience by fixing the optional dependency import issue, enhancing error messages with actionable suggestions, and adding comprehensive performance documentation. The automated release workflow streamlines the release process, and performance benchmarking in CI helps prevent performance regressions.
Migration Notes:
FairnessToolkitError)Critical Bug Fix: Fixed IndexError when using categorical Series with NaN values in intersectional analysis. The _intersectional_prep() function now properly handles categorical Series conversion by converting to string first, then to numpy array with proper NaN handling. This resolves issues in demographic_parity_difference(), equalized_odds_difference(), and mae_parity_difference() methods when using intersectional analysis with categorical data.
CLI Test Reliability: Fixed CLI tests to properly handle SystemExit exceptions from argparse, improving test reliability.
AB Test Assertions: Updated AB test assertions to account for bootstrap sampling variability, reducing false test failures.
Edge Case Handling: Fixed intersectional tests to handle empty DataFrame edge cases and resolved variable name conflicts in CLI command tests.
TEST_REVIEW_REPORT.md with comprehensive test suite significance analysis, including detailed evaluation of 9 major test suites with star ratings and insights on test suite prioritization.This patch release addresses a critical bug in intersectional analysis that could cause failures with categorical data, and significantly improves test suite reliability and documentation. The test suite now provides comprehensive coverage with all tests passing, ensuring the toolkit’s reliability and correctness.
Integrated End-to-End Workflow: Introduced unified three-step workflow orchestrator combining Measurement, Pipeline, and Training modules into a single automated process.
fairpipe run-pipeline executes complete workflow:
Extended Config Schema: Added training, fairness_metric, and validation_threshold fields to support integrated workflow. Config files can now specify training method (reductions, regularized, lagrangian) with method-specific parameters.
Complete MLflow Integration: Enhanced MLflow logger to log complete workflow results including baseline/final metrics, validation status, model artifacts, and config.yml. Enables tracking of fairness experiments over time.
demo_integrated.ipynb demonstrating the complete integrated workflow from raw data to validated model.Config System: Extended PipelineConfig to support training method selection with method-specific parameters. Configs without a training section continue to work for pipeline-only execution via fairpipe pipeline command.
Orchestrator: Enhanced to handle sensitive attribute encoding for PyTorch models and proper feature matrix construction. Sensitive attributes are now automatically excluded from feature matrices before training.
Result Object Handling: Validation function now correctly handles both Result objects and dict formats, improving compatibility.
Feature Matrix Construction: Sensitive attributes are now properly excluded from feature matrices before training, preventing data leakage.
Sensitive Attribute Encoding: String sensitive attributes are automatically encoded as integers for PyTorch models, preventing type errors.
Result Object Handling: Validation function now correctly handles both Result objects and dict formats.
This major update transforms the toolkit from modular components into a unified, integrated system. Users can now execute a complete fairness workflow from raw data to validated model with a single command, with automatic baseline comparison and threshold validation. This enables CI/CD integration and automated fairness assurance.
Migration Notes:
training section continue to work with fairpipe pipeline commandtraining section to config and use fairpipe run-pipeline commandRealTimeFairnessTracker: Fixed to use DatetimeIndex instead of timestamp column, ensuring proper time-series format as required. Metrics are now stored with timestamp as the index, improving time-series analysis capabilities.
FairnessDriftAndAlertEngine: Enhanced alert severity scoring to incorporate group size (n) from metrics. Smaller groups now reduce confidence in drift detection, preventing false alarms from statistically unreliable samples.
FairnessReportingDashboard: Updated to handle DatetimeIndex format with backward compatibility for timestamp column format.
FairnessReportingDashboard: Converted intersectional visualization from bar chart to heatmap (go.Heatmap) for better visualization of fairness metrics across intersectional subgroups. Heatmap uses diverging colormap (RdYlBu_r) to highlight disparities.
DatetimeIndex, maintaining compatibility with new format.demo_monitoring.ipynb that simulates a production stream and demonstrates all monitoring components working together:
tests/monitoring/test_tracker.py: Tests for tracker DatetimeIndex usage, CSV persistence, sliding window, and metric computationtests/monitoring/test_dashboard_and_drift.py: Tests for DatetimeIndex handling, heatmap visualization, and severity scoring with group sizeThis update addresses critical gaps identified in the Monitoring Module assessment, ensuring proper time-series format with DatetimeIndex, complete alert prioritization logic including group size, comprehensive demo notebook demonstrating all components, and improved visualization with heatmap for intersectional analysis.
Migration Notes:
DatetimeIndex format in metrics CSV filesT parameter not being passed to ExponentiatedGradient. The parameter is now correctly forwarded as max_iter to control iteration limits.sweep_pareto() to automatically save plots when save_path is provided, streamlining the workflow for generating and saving Pareto frontier visualizations.demo_training.ipynb providing comprehensive examples demonstrating all Training Module components with synthetic data generation, visualizations, and usage patterns.This update addresses critical gaps identified in the Training Module assessment, ensuring all components are properly documented, tested, and functional. The ReductionsWrapper fix ensures proper iteration control, and the expanded test coverage improves reliability.
Migration Notes:
T parameter for iteration limitsfairlearn.reductions.ExponentiatedGradient for training under fairness constraints (e.g., Demographic Parity).fairpipe train-regularized: Train NN with fairness regularizer and generate Pareto frontierfairpipe train-lagrangian: Train NN with Lagrangian fairness constraintsfairpipe calibrate: Apply group-specific calibration to prediction scores[training] extra for PyTorch and related dependencies.Unified CLI Configuration: Refined CLI configuration and profile loading (pipeline.config.yml) to support both pipeline and training profiles.
Exception Handling: Refined exception handling for ExponentiatedGradient compatibility and PyTorch gradient tracking.
tests/training/ for sklearn, torch, postproc, and visualization submodules.Phase 6 extends the toolkit’s capabilities beyond data-level fairness by embedding fairness constraints directly into model training workflows, ensuring equitable outcomes by design.
Migration Notes:
pip install -e .[training] to enable PyTorch dependenciesSystem Test: End-to-end CLI test (tests/system/test_cli_e2e_pipeline.py) verifying full pipeline execution and artifact generation.
Demo Notebook Generator: scripts/make_demo_notebook.py programmatically creates a clean, runnable demo.ipynb showing detection → mitigation → reporting.
Artifacts: Auto-generated demo.ipynb ready for Jupyter or VS Code use.
Documentation: Expanded README with Phase 5 instructions (E2E tests, demo generation, and MLflow logging).
Test Reliability: Improved test reliability for pipeline and detector integration.
Phase 5 finalized the first release candidate by validating the entire fairness pipeline through automated tests and a reproducible demo.
fairness-pipeline-dev-toolkit to fairpipe for improved user experiencefairness-pipeline-dev-toolkit to fairpipe. Users must update installation commands, but no code changes are required (Python imports remain unchanged).training section continue to work with fairpipe pipeline command.DatetimeIndex format, but backward compatibility maintained for timestamp columns.No deprecations in current version.
Note: Dates marked as “2025-01-XX” are placeholders. Update with actual release dates when known.