1 INTRODUCTION  
1.1 Related Surveys  
1.2 Structure of the Survey  

2 PRELIMINARIES  

3 GRADIENT DESCENT OPTIMIZATION ALGORITHMS  
3.1 Gradient Descent Variants  
3.2 Momentum  
3.3 Adaptive Gradient Algorithms  

4 LARGE BATCH TRAINING  
4.1 Large Batch Training Difficulties  
4.2 Learning Rate Scaling for Large Batch  
4.3 Adaptive Layerwise Learning  
4.4 Adaptive Batch Size  
4.5 Efficient Scaling  

5 GENERALIZATION GAP  
5.1 Sharp and Flat (Wide) Minima  
5.2 Generalization Gap and Sharp Minima  
5.3 Gradient Noise Ratio  
5.4 Train Longer, Generalize Better  

6 SECOND ORDER OPTIMIZATION  
6.1 Second-Order Optimization Basics  
6.2 Newton’s Method  
6.3 Hessian-Free Method  
6.4 K-FAC  
6.5 Shampoo  

7 COMMUNICATION  
7.1 Gradient Compression  
7.2 Reducing Communication Frequency  

8 MEMORY  
8.1 Mix-Precision Training  
8.2 Memory Efficient Adaptive Optimization  

9 CONCLUSION