Fairness Metrics (#5093)

* Added three definitions of fairness * Updated CHANGELOG * Added DemographicParityWithoutGroundTruth and finished tests * finished refactoring Independence, Separation, and Sufficiency to accumulate * added distributed functionality to Independence, Sufficiency, and Separation * Finished aggregate and distributed functionality for DemographicParityWithoutGroundTruth * fixed GPU and doc issues * fixed GPU and doc issues * fixed GPU and doc issues * fixed GPU issues * fixed GPU issues * added init file * fixed typo * minor docstring changes * minor changes to docstring * Added simple explanations of fairness metrics to docstrings * Further vectorized all metric implementations * Fixed device issue Co-authored-by: Arjun Subramonian <[email protected]> Co-authored-by: Akshita Bhagia <[email protected]> Co-authored-by: Dirk Groeneveld <[email protected]>
allenai · Apr 20, 2021 · f877fdc · f877fdc
1 parent f732b41
commit f877fdc
Show file tree

Hide file tree

Showing 6 changed files with 1,348 additions and 2 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Add new dimension to the `interpret` module: influence functions via the `InfluenceInterpreter` base class, along with a concrete implementation: `SimpleInfluence`.
 - Added a `quiet` parameter to the `MultiProcessDataLoading` that disables `Tqdm` progress bars.
 - The test for distributed metrics now takes a parameter specifying how often you want to run it.
+- Created the fairness module and added four fairness metrics: `Independence`, `Separation`, `Sufficiency`, and `DemographicParityWithoutGroundTruth`.
 
 ### Changed
 
@@ -30,7 +31,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- Ported the following HuggingFace `LambdaLR`-based schedulers: `ConstantLearningRateScheduler`, `ConstantWithWarmupLearningRateScheduler`, `CosineWithWarmupLearningRateScheduler`, `CosineHardRestartsWithWarmupLearningRateScheduler`.
+- Ported the following Huggingface `LambdaLR`-based schedulers: `ConstantLearningRateScheduler`, `ConstantWithWarmupLearningRateScheduler`, `CosineWithWarmupLearningRateScheduler`, `CosineHardRestartsWithWarmupLearningRateScheduler`.
 - Added new `sub_token_mode` parameter to `pretrained_transformer_mismatched_embedder` class to support first sub-token embedding
 - Added a way to run a multi task model with a dataset reader as part of `allennlp predict`.
 - Added new `eval_mode` in `PretrainedTransformerEmbedder`. If it is set to `True`, the transformer is _always_ run in evaluation mode, which, e.g., disables dropout and does not update batch normalization statistics.

diff --git a/allennlp/common/testing/__init__.py b/allennlp/common/testing/__init__.py
@@ -64,7 +64,13 @@ def assert_metrics_values(
     atol: float = 1e-05,
 ):
     for key in metrics:
-        assert_allclose(metrics[key], desired_values[key], rtol=rtol, atol=atol)
+        if isinstance(metrics[key], Dict) and isinstance(desired_values[key], Dict):
+            for subkey in metrics[key]:
+                assert_allclose(
+                    metrics[key][subkey], desired_values[key][subkey], rtol=rtol, atol=atol
+                )
+        else:
+            assert_allclose(metrics[key], desired_values[key], rtol=rtol, atol=atol)
 
 
 def global_distributed_metric(

diff --git a/allennlp/fairness/__init__.py b/allennlp/fairness/__init__.py
@@ -0,0 +1,14 @@
+"""
+This module contains tools to:
+
+1. measure the fairness of models according to multiple definitions of fairness
+2. measure bias amplification
+3. debias embeddings during training time and post-processing
+"""
+
+from allennlp.fairness.fairness_metrics import (
+    Independence,
+    Separation,
+    Sufficiency,
+    DemographicParityWithoutGroundTruth,
+)