Skip to content

Latest commit

 

History

History

model_card_gen

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 

Model Card Generator

Model Card Generator allows users to create interactive HTML reports of containing model performance and fairness metrics

Model Card Sections

Section
Subsection Decription
Model Details Overview A brief, one-line description of the model card.
Documentation A thorough description of the model and its usage.
Owners The individuals or teams who own the model.
Version The version of the schema
Licenses The model's license for use.
References Links providing more information about the model.
Citations How to reference this model card.
Path The path where the model is stored.
Graphics Collection of overview graphics.
Model Parameters Model Architecture The architecture of the model.
Data The datasets used to train and evaluate the model.
Input Format The data format for inputs to the model.
Input Format Map The data format for inputs to the model, in key-value format.
Output Format The data format for outputs from the model.
Output Format Map The data format for outputs from the model, in key-value format.
Quantitative analysis Performance Metrics The model performance metrics being reported.
Graphics Colleciton of performance graphics
Considerations Users Who are the intended users of the model?
Use Cases What are the intended use cases of the model?
Limitations What are the known technical limitations of the model? E.g. What kind(s) of data should the model be expected not to perform well on? What are the factors that might degrade model performance?
Tradeoffs What are the known tradeoffs in accuracy/performance of the model?
Ethical Considerations What are the ethical (or environmental) risks involved in the application of this model?

Install

Step 1: Clone the GitHub repository.

git clone https://github.com/Intel/intel-xai-tools.git

Step 2: Navigate to intel-xai-tools directory.

cd intel-xai-tools/

Step 3: Install package with pip.

pip install .

Run

Model Card Generator Inputs

The ModelCardGen.generate classmethod requires three inputs and returns a ModelCardGen class instance:

  • data_sets (dict) : dictionary containing the user-defined name of the dataset as key and the path to the tfrecords or raw dataframe containing prediction values as value.

    • For TensorFlow TFRecords {'eval': TensorflowDataset(dataset_path='eval.tfrecord*')} (file glob pattern)
    • For PyTorch Dataset {'eval': PytorchDataset(pytorch_dataset, feature_names=feature_names)}
    • For Pandas DataFrames {'eval': pd.Daraframe({"y_true": y_true, "y_pred": ypred})}
  • model_path (str) : this field represents the path to the TensorFlow SavedModel and it is only required for TensorFlow models.

  • eval_config (tfma.EvalConfig or str) : this is either the path to the proto config file used by the tfma evaluator or the proto string to be parsed. For example, let us review the following file entitled "eval_config.proto" defined for the COMPAS proxy model found in /notebooks/model_card_gen/compas_with_model_card_gen/compas-model-card-tfx.ipynb.

TFMA EvalConfig

For example eval_config parameter, let us review the following file entitled "eval_config.proto" defined for the COMPAS proxy model found in /notebooks/model_card_gen/compas_with_model_card_gen/compas-model-card-tfx.ipynb.

In the model_specs section it tells the evaluator "label_key" is the ground truth label. In the metric_specs section it defines the following metrics to be computed: "BinaryAccuracy", "AUC", "ConfusionMatrixPlot", and "FairnessIndicators". In the slicing_specs section it tells the evaluator to compute these metrics accross all datapoints and aggregate these metrics grouped by the "race" feature.

model_specs {
    label_key: 'is_recid'
  }
metrics_specs {
  metrics {class_name: "BinaryAccuracy"}
  metrics {class_name: "AUC"}
  metrics {class_name: "ConfusionMatrixPlot"}
  metrics {
    class_name: "FairnessIndicators"
      config: '{"thresholds": [0.25, 0.5, 0.75]}'
  }
  }
# The overall slice
slicing_specs {}
slicing_specs {
    feature_keys: 'race'
    }
options {
    include_default_metrics { value: false }
  }

If we are computing metrics on a raw dataframe we must add the "prediction_key" to model_specs as follows

model_specs {
    label_key: 'y_true'
    prediction_key: 'y_pred'
  }
...

Populate Model Card user-defined fields

The Model Card object that is generated can be serialized/deserialized via the JSON schema defined in schema/v*/model_card.schema.json. You can use a Python dictionary to provide content to static fields like those contained in the "model_details" section of the mc variable below. Any field can be added to this dictionary of pre-defined fields as long as it is it coheres to the schema being used.

mc = {
  "model_details": {
    "name": "COMPAS (Correctional Offender Management Profiling for Alternative Sanctions)",
    "overview": "COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a public dataset, which contains approximately 18,000 criminal cases from Broward County, Florida between January, 2013 and December, 2014. The data contains information about 11,000 unique defendants, including criminal history demographics, and a risk score intended to represent the defendant’s likelihood of reoffending (recidivism)",
    "owners": [
      {
        "name": "Intel XAI Team",
        "contact": "[email protected]"
      }
    ],
    "references": [
      {
        "reference": "Wadsworth, C., Vera, F., Piech, C. (2017). Achieving Fairness Through Adversarial Learning: an Application to Recidivism Prediction. https://arxiv.org/abs/1807.00199."
      },
      {
        "reference": "Chouldechova, A., G'Sell, M., (2017). Fairer and more accurate, but for whom? https://arxiv.org/abs/1707.00046."
      },
      {
        "reference": "Berk et al., (2017), Fairness in Criminal Justice Risk Assessments: The State of the Art, https://arxiv.org/abs/1703.09207."
      }
    ],
    "graphics": {
      "description": " "
    }
  },
  "quantitative_analysis": {
    "graphics": {
      "description": " "
    }
  },
  "schema_version": "0.0.1"
}

A more comprehensive JSON example, that includes formatting for Ethical Considerations, can be found here: /model_card_gen/intel_ai_safety/model_card_gen/docs/examples/json/model_card_example.json.

Create Model Card

from intel_ai_safety.model_card_gen.model_card_gen import ModelCardGen

model_path = 'compas/model'
data_paths = {
  'eval': 'compas/eval.tfrecord',
  'train': 'compas/train.tfrecord'
}
eval_config = 'compas/eval_config.proto'

mcg = ModelCardGen.generate(_data_paths, _model_path, _eval_config, model_card=mc)

Test

Step 1: Test by installing test dependencies:

pip install ".[test]"

Step 2: Run tests

python -m pytest tests/

Markers

The following custom markers have been defined in the Model Card Generator tests:

@pytest.mark.tensorflow: test requires tensorflow to be installed

@pytest.mark.pytorch: test requires pytorch and tensorflow-model-analysis to be installed

@pytest.mark.common: test does not require a specific framework to be installed

Note that running PyTorch tests still requires TensorFlow libararies for model analysis.

Sample test commands using markers

Run only the TensorFlow tests:

python -m pytest tests/ -m tensorflow

Run the PyTorch and common tests:

python -m pytest tests/ -m "pytorch or common"