From c555fd8f0606a6fdb8ed1fed6cb7c12e0d303948 Mon Sep 17 00:00:00 2001 From: Jonas Mueller <1390638+jwmueller@users.noreply.github.com> Date: Fri, 27 Sep 2024 15:31:03 -0700 Subject: [PATCH] reorder readme --- README.md | 54 +++++++++++++++++++++++++++--------------------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index dae03f8..e66ec76 100644 --- a/README.md +++ b/README.md @@ -6,33 +6,33 @@ To quickly learn how to run cleanlab on your own data, first check out the [quic ## Table of Contents -| | Example | Description | -| --- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 1 | [datalab](datalab_image_classification/README.md) | Use Datalab to detect various types of data issues in (a subset of) the Caltech-256 image classification dataset. | -| 2 | [find_label_errors_iris](find_label_errors_iris/find_label_errors_iris.ipynb) | Find label errors introduced into the Iris classification dataset. | -| 3 | [classifier_comparison](classifier_comparison/classifier_comparison.ipynb) | Use CleanLearning to train 10 different classifiers on 4 dataset distributions with label errors. | -| 4 | [hyperparameter_optimization](hyperparameter_optimization/hyperparameter_optimization.ipynb) | Hyperparameter optimization to find the best settings of CleanLearning's optional parameters. | -| 5 | [simplifying_confident_learning](simplifying_confident_learning/simplifying_confident_learning.ipynb) | Straightforward implementation of Confident Learning algorithm with raw numpy code. | -| 6 | [visualizing_confident_learning](visualizing_confident_learning/visualizing_confident_learning.ipynb) | See how cleanlab estimates parameters of the label error distribution (noise matrix). | -| 7 | [find_tabular_errors](find_tabular_errors/find_tabular_errors.ipynb) | Handle mislabeled [tabular data](https://github.com/cleanlab/s/blob/master/student-grades-demo.csv) to improve a XGBoost classifier. | -| 8 | [fine_tune_LLM](fine_tune_LLM/LLM_with_noisy_labels_cleanlab.ipynb) | Fine-tuning OpenAI language models with noisily labeled text data | -| 9 | [cnn_mnist](cnn_mnist/find_label_errors_cnn_mnist.ipynb) | Finding label errors in MNIST image data with a [Convolutional Neural Network](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/experimental/mnist_pytorch.py). | -| 10 | [huggingface_keras_imdb](huggingface_keras_imdb/huggingface_keras_imdb.ipynb) | CleanLearning for text classification with Keras Model + pretrained BERT backbone and Tensorflow Dataset. | -| 11 | [fasttext_amazon_reviews](fasttext_amazon_reviews/fasttext_amazon_reviews.ipynb) | Finding label errors in Amazon Reviews text dataset using a cleanlab-compatible [FastText model](fasttext_amazon_reviews/fasttext_wrapper.py). | -| 12 | [multiannotator_cifar10](multiannotator_cifar10/multiannotator_cifar10.ipynb) | Iteratively improve consensus labels and trained classifier from data labeled by multiple annotators. | -| 13 | [llm_evals_w_crowdlab](llm_evals_w_crowdlab/llm_evals_w_crowdlab.ipynb) | Reliable LLM Evaluation with multiple human/AI reviewers of varying competency (via CROWDLAB and LLM-as-judge GPT token probabilities). | -| 14 | [active_learning_multiannotator](active_learning_multiannotator/active_learning.ipynb) | Improve a classifier model by iteratively collecting additional labels from data annotators. This active learning pipeline considers data labeled in batches by multiple (imperfect) annotators. | -| 15 | [active_learning_single_annotator](active_learning_single_annotator/active_learning_single_annotator.ipynb) | Improve a classifier model by iteratively labeling batches of currently-unlabeled data. This demonstrates a standard active learning pipeline with *at most one label* collected for each example (unlike our multi-annotator active learning notebook which allows re-labeling). | -| 16 | [active_learning_transformers](active_learning_transformers/active_learning.ipynb) | Improve a Transformer model for classifying politeness of text by iteratively labeling and re-labeling batches of data using multiple annotators. If you haven't done active learning with re-labeling, try the [active_learning_multiannotator](active_learning_multiannotator/active_learning.ipynb) notebook first. | -| 17 | [outlier_detection_cifar10](outlier_detection_cifar10/outlier_detection_cifar10.ipynb) | Train AutoML for image classification and use it to detect out-of-distribution images. | -| 18 | [multilabel_classification](multilabel_classification/image_tagging.ipynb) | Find label errors in an image tagging dataset ([CelebA](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html)) using a [Pytorch model](multilabel_classification/pytorch_network_training.ipynb) you can easily train for multi-label classification. | -| 19 | [entity_recognition](entity_recognition/) | Train Transformer model for Named Entity Recognition and produce out-of-sample `pred_probs` for **cleanlab.token_classification**. | -| 20 | [transformer_sklearn](transformer_sklearn/transformer_sklearn.ipynb) | How to use `KerasWrapperModel` to make any Keras model sklearn-compatible, demonstrated here for a BERT Transformer. | -| 21 | [cnn_coteaching_cifar10](cnn_coteaching_cifar10/README.md) | Train a [Convolutional Neural Network](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/experimental/cifar_cnn.py) on noisily labeled Cifar10 image data using cleanlab with [coteaching](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/experimental/coteaching.py). | -| 22 | [non_iid_detection](non_iid_detection/non_iid_detection.ipynb) | Use Datalab to detect non-IID sampling (e.g. drift) in datasets based on numeric features or embeddings. | -| 23 | [object_detection](object_detection/README.md) | Train Detectron2 object detection model for use with cleanlab. | -| 24 | [semantic segmentation](segmentation/training_ResNeXt50_for_Semantic_Segmentation_on_SYNTHIA.ipynb) | Train ResNeXt semantic segmentation model for use with cleanlab. | -| 24 | [spurious correlations](spurious_correlations_datalab/detecting_spurious_correlations.ipynb) | Train a CNN model on spurious and non-spurious versions of a subset of [Food-101](https://data.vision.ee.ethz.ch/cvl/datasets_extra/food-101/) dataset. Use `Datalab` to detect issues in the spuriously correlated datasets. | +| Example | Description | +| ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [datalab](datalab_image_classification/README.md) | Use Datalab to detect various types of data issues in (a subset of) the Caltech-256 image classification dataset. | +| [llm_evals_w_crowdlab](llm_evals_w_crowdlab/llm_evals_w_crowdlab.ipynb) | Reliable LLM Evaluation with multiple human/AI reviewers of varying competency (via CROWDLAB and LLM-as-judge GPT token probabilities). | +| [fine_tune_LLM](fine_tune_LLM/LLM_with_noisy_labels_cleanlab.ipynb) | Fine-tuning OpenAI language models with noisily labeled text data | +| [entity_recognition](entity_recognition/) | Train Transformer model for Named Entity Recognition and produce out-of-sample `pred_probs` for **cleanlab.token_classification**. | +| [multiannotator_cifar10](multiannotator_cifar10/multiannotator_cifar10.ipynb) | Iteratively improve consensus labels and trained classifier from data labeled by multiple annotators. | +| [active_learning_multiannotator](active_learning_multiannotator/active_learning.ipynb) | Improve a classifier model by iteratively collecting additional labels from data annotators. This active learning pipeline considers data labeled in batches by multiple (imperfect) annotators. | +| [active_learning_single_annotator](active_learning_single_annotator/active_learning_single_annotator.ipynb) | Improve a classifier model by iteratively labeling batches of currently-unlabeled data. This demonstrates a standard active learning pipeline with *at most one label* collected for each example (unlike our multi-annotator active learning notebook which allows re-labeling). | +| [active_learning_transformers](active_learning_transformers/active_learning.ipynb) | Improve a Transformer model for classifying politeness of text by iteratively labeling and re-labeling batches of data using multiple annotators. If you haven't done active learning with re-labeling, try the [active_learning_multiannotator](active_learning_multiannotator/active_learning.ipynb) notebook first. | +| [outlier_detection_cifar10](outlier_detection_cifar10/outlier_detection_cifar10.ipynb) | Train AutoML for image classification and use it to detect out-of-distribution images. | +| [multilabel_classification](multilabel_classification/image_tagging.ipynb) | Find label errors in an image tagging dataset ([CelebA](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html)) using a [Pytorch model](multilabel_classification/pytorch_network_training.ipynb) you can easily train for multi-label classification. | +| [find_label_errors_iris](find_label_errors_iris/find_label_errors_iris.ipynb) | Find label errors introduced into the Iris classification dataset. | +| [classifier_comparison](classifier_comparison/classifier_comparison.ipynb) | Use CleanLearning to train 10 different classifiers on 4 dataset distributions with label errors. | +| [hyperparameter_optimization](hyperparameter_optimization/hyperparameter_optimization.ipynb) | Hyperparameter optimization to find the best settings of CleanLearning's optional parameters. | +| [simplifying_confident_learning](simplifying_confident_learning/simplifying_confident_learning.ipynb) | Straightforward implementation of Confident Learning algorithm with raw numpy code. | +| [visualizing_confident_learning](visualizing_confident_learning/visualizing_confident_learning.ipynb) | See how cleanlab estimates parameters of the label error distribution (noise matrix). | +| [find_tabular_errors](find_tabular_errors/find_tabular_errors.ipynb) | Handle mislabeled [tabular data](https://github.com/cleanlab/s/blob/master/student-grades-demo.csv) to improve a XGBoost classifier. | +| [cnn_mnist](cnn_mnist/find_label_errors_cnn_mnist.ipynb) | Finding label errors in MNIST image data with a [Convolutional Neural Network](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/experimental/mnist_pytorch.py). | +| [huggingface_keras_imdb](huggingface_keras_imdb/huggingface_keras_imdb.ipynb) | CleanLearning for text classification with Keras Model + pretrained BERT backbone and Tensorflow Dataset. | +| [fasttext_amazon_reviews](fasttext_amazon_reviews/fasttext_amazon_reviews.ipynb) | Finding label errors in Amazon Reviews text dataset using a cleanlab-compatible [FastText model](fasttext_amazon_reviews/fasttext_wrapper.py). | +| [transformer_sklearn](transformer_sklearn/transformer_sklearn.ipynb) | How to use `KerasWrapperModel` to make any Keras model sklearn-compatible, demonstrated here for a BERT Transformer. | +| [cnn_coteaching_cifar10](cnn_coteaching_cifar10/README.md) | Train a [Convolutional Neural Network](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/experimental/cifar_cnn.py) on noisily labeled Cifar10 image data using cleanlab with [coteaching](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/experimental/coteaching.py). | +| [non_iid_detection](non_iid_detection/non_iid_detection.ipynb) | Use Datalab to detect non-IID sampling (e.g. drift) in datasets based on numeric features or embeddings. | +| [object_detection](object_detection/README.md) | Train Detectron2 object detection model for use with cleanlab. | +| [semantic segmentation](segmentation/training_ResNeXt50_for_Semantic_Segmentation_on_SYNTHIA.ipynb) | Train ResNeXt semantic segmentation model for use with cleanlab. | +| [spurious correlations](spurious_correlations_datalab/detecting_spurious_correlations.ipynb) | Train a CNN model on spurious and non-spurious versions of a subset of [Food-101](https://data.vision.ee.ethz.ch/cvl/datasets_extra/food-101/) dataset. Use `Datalab` to detect issues in the spuriously correlated datasets. | ## Instructions