1 INTRODUCTION

2 PRELIMINARIES OF MACHINE LEARNING

3 MACHINE LEARNING TESTING
3.1 Definition
3.2 ML Testing Workflow
3.3 ML Testing Components
3.4 ML Testing Properties
3.5 Software Testing vs. ML Testing

4 PAPER COLLECTION AND REVIEW SCHEMA
4.1 Survey Scope
4.2 Paper Collection Methodology
4.3 Collection Results
4.4 Paper Organisation

5 ML TESTING WORKFLOW
5.1 Test Input Generation
5.2 Test Oracle
5.3 Test Adequacy
5.4 Test Prioritisation and Reduction
5.5 Bug Report Analysis
5.6 Debug and Repair
5.7 General Testing Framework and Tools

6 ML PROPERTIES TO BE TESTED
6.1 Correctness
6.2 Model Relevance
6.3 Robustness and Security
6.4 Efficiency
6.5 Fairness
6.6 Interpretability
6.7 Privacy

7 ML TESTING COMPONENTS
7.1 Bug Detection in Data
7.2 Bug Detection in Learning Program
7.3 Bug Detection in Framework

8 APPLICATION SCENARIOS
8.1 Autonomous Driving
8.2 Machine Translation
8.3 Natural Language Inference

9 ANALYSIS OF LITERATURE REVIEW
9.1 Timeline
9.2 Research Distribution among Machine Learning Categories
9.3 Research Distribution among Different Testing Properties
9.4 Datasets Used in ML Testing
9.5 Open-source Tool Support in ML Testing

10 CHALLENGES AND OPPORTUNITIES
10.1 Challenges in ML Testing
10.2 Research Opportunities in ML testing

11 CONCLUSION