1 Introduction  
1.1 Certification  
1.2 Explanation  
1.3 Organisation of This Survey  

2 Preliminaries  
2.1 Deep Neural Networks  
2.2 Verification  
2.3 Testing  
2.4 Interpretability  
2.5 Distance Metric and $d$-Neighbourhood  

3 Safety Problems and Safety Properties  
3.1 Adversarial Examples  
3.2 Local Robustness Property  
3.3 Output Reachability Property  
3.4 Interval Property  
3.5 Lipschitzian Property  
3.6 Relationship between Properties  
3.7 Instance-wise Interpretability  

4 Verification  
4.1 Approaches with Deterministic Guarantees  
4.2 Approaches to Compute an Approximate Bound  
4.3 Approaches to Compute Converging Bounds  
4.4 Approaches with Statistical Guarantees  
4.5 Computational Complexity of Verification  
4.6 Summary  

5 Testing  
5.1 Coverage Criteria for DNNs  
5.2 Test Case Generation  
5.3 Model-Level Mutation Testing  
5.4 Summary  

6 Adversarial Attack and Defence  
6.1 Adversarial Attacks  
6.2 Adversarial Attacks by Natural Transformations  
6.3 Input-Agnostic Adversarial Attacks  
6.4 Other Types of Universal Adversarial Perturbations  
6.5 Summary of Adversarial Attack Techniques  
6.6 Adversarial Defence  
6.7 Certified Adversarial Defence  
6.8 Summary of Adversarial Defence Techniques  

7 Interpretability  
7.1 Instance-wise Explanation by Visualising a Synthesised Input  
7.2 Instance-wise Explanation by Ranking  
7.3 Instance-wise Explanation by Saliency Maps  
7.4 Model Explanation by Influence Functions  
7.5 Model Explanation by Simpler Models  
7.6 Information-flow Explanation by Information Theoretical Methods  
7.7 Summary  

8 Future Challenges  
8.1 Distance Metrics closer to Human Perception  
8.2 Improvement to Robustness  
8.3 Other Machine Learning Models  
8.4 Verification Completeness  
8.5 Scalable Verification with Tighter Bounds  
8.6 Validation of Testing Approaches  
8.7 Learning-Enabled Systems  
8.8 Distributional Shift, Out-of-Distribution Detection, and Run-time Monitoring  
8.9 Unifying Formulation of Interpretability  
8.10 Application of Interpretability to other Tasks  
8.11 Human-in-the-Loop  

9 Conclusions