README.org

Formal Conceptual Views in Neural Networks

This is the repository to the paper Formal Conceptual Views in Neural Networks.

With the present work, we introduce two notions for conceptual views of a neural network, specifically a many-valued and a symbolic view. Both provide novel analysis methods to enable a human AI analyst to grasp deeper insights into the knowledge that is captured by the neurons of a network.

We test the expressivity of our novel views through different experiments on the ImageNet and Fruit-360 data set. Furthermore, we show to which extent the views allow to quantify the conceptual similarity of different learning architectures. Finally, we demonstrate how conceptual views can be applied for abductive learning of human comprehensible rules from neurons. In summary, with our work, we contribute to the most relevant task of globally explaining neural networks models.

Table of Contents

• Requirements: Python and Clojure
• Data: ImageNet and Fruit-360
• Setup
  • Compute ImageNet Many-Valued and Symbolic Conceptual Views
  • Compute Fruit-360 Many-Valued and Symbolic Conceptual Views
• Evaluate Many-Valued Conceptual View
  • Statistics on the Many-Valued Views
  • Fidelity of Many-Valued Conceptual views
  • Similarity of Models using Many-Valued conceptual views
• Symbolic Conceptual Views
  • Ablation Study: Activation Function and Number of Neurons
  • Fidelity of Symbolic Conceptual Views
  • Similarity of and Zoom in on Symbolic Conceptual Views using FCA

Requirements

This project uses functionality from python for the machine learning parts and the conexp-clj framework for the methods from formal concept analysis.

Python

The used packages and versions can be found in requirements.txt. We used Python version 3.7.3.

pip install -r requirements.txt

Clojure

We used Clojure version 1.10.1 and conexp-clj version 2.3.0. There are two options to evaluate the Clojure code. The first is to build the most recent version from the repository using Leiningen.

git clone https://github.com/tomhanika/conexp-clj
cd conexp-clj
lein uberjar

A standalone jar can then be found at /builds/uberjar/conexp-clj-VERSION-SNAPSHOT-standalone.jar and executing it will start a REPL which can be used to execute the clojure code.

java -jar /builds/uberjar/conexp-clj-VERSION-SNAPSHOT-standalone.jar

Alternatively, a recent executable can be downloaded from the Maven repository.

Data

There are three data sources that we use. The first is the Fruit-360 data set which can be downloaded using the init-data.sh skript. The data set is then extracted into the image-data/fruit360 directory.

The second data set is the ImageNet data set from the visual recognition challenge. We only use its test set which should be extracted to image-data/imagenet/test.

Setup

Compute ImageNet Many-Valued and Symbolic Conceptual Views

First, we need to compute the many-valued and symbolic conceptual views for the ImageNet data set. The code can be found in imagenet_conceptual_views.org. This results in a representation of the objects and classes in a single pseudo metric space.

python src/tangled/imagenet_conceptual_views.py

Compute Fruit-360 Many-Valued and Symbolic Conceptual Views

Next, we need to compute the many-valued and symbolic conceptual views for the Fruit-360 data set. The model files were split using the Linux command split. The code can be found in fruit_conceptual_views.org

python src/tangled/fruit_conceptual_views.py

The code which we used to train these models is located in train_fruits.org.

Evaluate Many-Valued Conceptual View

Statistics

Statistics on the ImageNet models can be computed using statistics.org.

python src/tangled/statistics.py

Experiment: Quality by Fidelity

We evaluated the quality of the many-valued conceptual views using the fidelity of a simple one nearest neighbor classifier in the pseudo-metric space and the original model.

python src/tangled/fidelity.py

Fidelity scores for the ImageNet models:

Fidelity scores for the Fruit-360 models:

Experiment: ImageNet Model Similarity

The pseudo metric space allows for comparing models using Gromov-Wasserstein distance. We compare the resulting similarities using a pairwise fidelity of the original models.

python src/tangled/imagenet_similarity.py

Symbolic Conceptual View

Ablation Study for Number of Neurons and Choice of Activation Function

We conducted an ablation study for the influence of the activation function and the number of neurons. We did ten training runs of the same architecture for each parameter setting.

python src/tangled/ablation.py

We evaluated their results using the fidelity of the views.

And the shape of the views where we identified the tanh activation function to cause the clearest visible separation between negative and positive values and highest fidelity scores.

Experiment: Quality by Fidelity

We evaluated the quality of the symbolic conceptual views using the fidelity of a simple one nearest neighbor classifier using the symbolic views and the original model.

python src/tangled/fidelity.py

Fidelity for the symbolic conceptual views on the ImageNet models:

Fidelity for the symbolic conceptual views on the Fruit-360 models:

Formal Concept Analysis

The code can be found in formal_conceptual_views.org and should be executed in order in a Clojure REPL.

The first result are the number of formal concepts.

Secondly, we can compute a similarity based on formal concepts. This similarity is based on concepts in which two fruits co-occur. For example the fruits Plum, Cherry, Apple Pink Lady and Apple Red in the VGG16 transfer learned model.

Using the formal concept analysis, we can zoom in individual fruits in the model and how it related other fruits in a hierarchical manner. For this, we employ the concept lattice.

Abductive learning of partial explanations

To derive explanations for the information captured by the neurons, we employ subgroup detection for visual and botanic taxon features. The code can be found in abductive_explanation.org.

python src/tangled/abductive_explanation.py

We provide explanations for neuron 13, as well as, representations for the apple taxon and orange color.