Harmonizing the object recognition strategies of deep neural networks with humans

Thomas Fel*, Ivan Felipe Rodriguez*, Drew Linsley*,
Thomas Serre
Carney Institute for Brain Science, Brown University, Providence, RI 02912
{thomas_fel,ivan_felipe_rodriguez,drew_linsley,thomas_serre}@brown.edu

Read the official paper »

Explore results · Github · Models zoo · Tutorials · Click-me paper

Paper summary

The many successes of deep neural networks (DNNs) over the past decade have largely been driven by computational scale rather than insights from biological intelligence. Here, we explore if these trends have also carried concomitant improvements in explaining visual strategies underlying human object recognition. We do this by comparing two related but distinct properties of visual strategies in humans and DNNs: where they believe important visual features are in images and how they use those features to categorize objects. Across 85 different DNNs and three independent datasets measuring human visual strategies on ImageNet, we find a trade-off between DNN top-1 categorization accuracy and their alignment with humans. State-of-the-art DNNs are progressively becoming less aligned with humans. We rectify this growing issue by introducing the harmonization procedure: a general-purpose training routine that aligns DNN and human visual strategies while improving object classification performance.

Aligning the Gradients

Human and DNNs rely on different features to recognize objects. In contrast, our neural harmonizer aligns DNN feature importance with humans. Gradients are smoothed from both humans and DNNs with a Gaussian kernel to improve visualization.

Breaking the trade-off between performance and alignment

The trade-off between DNN performance and alignment with human feature importance from the ClickMe dataset. Human feature alignment is the mean Spearman correlation between human and DNN feature importance maps, normalized by the average inter-rater alignment of humans. The grey-shaded region illustrates the convex hull of the trade-off between ImageNet accuracy and human feature alignment. All the models trained with the harmonization procedure are more accurate and aligned than versions of those models trained only for classification. Arrows denote a shift in performance after training with the harmonization procedure.

Authors

Thomas Fel*

Ivan Felipe Rodriguez*

Drew Linsley*

Thomas Serre

* : all authors have contributed equally.

🗞️ Citation

If you use or build on our work as part of your workflow in a scientific publication, please consider citing the official paper:

@article{fel2022aligning,
  title={Harmonizing the object recognition strategies of deep neural networks with humans},
  author={Fel, Thomas and Felipe, Ivan and Linsley, Drew and Serre, Thomas},
  journal={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2022}
}

Moreover, this paper relies heavily on previous work from the Lab, notably Learning What and Where to Attend where the ambitious ClickMe dataset was collected.

@article{linsley2018learning,
  title={Learning what and where to attend},
  author={Linsley, Drew and Shiebler, Dan and Eberhardt, Sven and Serre, Thomas},
  journal={International Conference on Learning Representations (ICLR)},
  year={2019}
}

Tutorials

Evaluate your own model (pytorch and tensorflow)

📝 License

The package is released under MIT license.