1 Introduction

2 Definitions

3 Neuron Analysis Methods  
3.1 Visualization  
3.2 Corpus-based Methods  
3.3 Probing-based Methods  
3.4 Causation-based Methods  
3.5 Miscellaneous Methods

4 Evaluation  
4.1 Ablation  
4.2 Classification Performance  
4.3 Information Theoretic Metric  
4.4 Concept Selectivity  
4.5 Qualitative Evaluation

5 Findings  
5.1 Concept Discovery  
5.2 Architectural Analysis  
5.3 Summary of Findings

6 Applications  
6.1 Controlling Model’s Behavior  
6.2 Model Distillation and Efficiency  
6.3 Domain Adaptation  
6.4 Compositional Explanations

7 Open Issues and Future Directions