LENS Method¶
The pages contain a lot of information, and the goal is to guide you gradually towards a better understanding of interpreting these results.
Summary¶
In summary, the method is quite simple: for each class, we take the points in the ImageNet dataset predicted as and extract the concepts (directions on intermediate space) from these points using the CRAFT method. Once these concepts are obtained, they are just vectors in the latent space, and it is possible to obtain a feature visualization - an image that maximise this direction - of these vectors using MACO . Finally, since the decomposition is performed on the last layer, there is a linear relationship between the concept embedding and the model's decision. More details on the estimation of importance can be found in the Holistic article. Now let's move on to interpreting the different components.
Understanding concept importance¶
For each point in the dataset, it is possible to obtain it's concepts values where each values represent a concept. Moreover, we can also compute an importance vector, this vector has a scalar value per concept, representing how crucial a concept was for the classification. Based on these vectors, we perform UMAP to obtain the clustering graph. Then, colors are assigned to each point based on its most important concept. If a concept is the most important for at least 10 times, it is considered important and will be colored. Finally, the global importance is the average of the logit scores contributed by the concepts across all points. Thus, a global importance of 30% means that the concept represents 30% of the sum of logits for the class.
Understanding Concept proximity¶
For each concept, it is possible to obtain its proximity to other concepts, i.e., how close they are in the latent space. To do this, we calculate the pairwise cosine similarity between the concepts. We then project the results into 2D. Thus, two close concepts will have a close cosine similarity.
Understanding Concept Alignment to Classes¶
Similar to concept proximity, it is possible to compare the class's separating hyperplane with the concept, which gives us a degree of alignment between the concept and each class. The reported score is also the cosine similarity between the two vectors.