1 INTRODUCTION  
1.1 Related Surveys  
1.2 Survey Organization  

2 FUNDAMENTALS OF DL AND DISTRIBUTED DL  
2.1 Deep Learning  
2.2 Distributed DL Parallelism Modes  
2.3 Resource Allocation and Workload Scheduling for Distributed DL  

3 RESOURCE ALLOCATION  

4 WORKLOAD SCHEDULING  

5 DISTRIBUTED TRAINING OF LLMs: A CASE STUDY  

6 CONCLUSION AND OUTLOOK  
6.1 Conclusion  
6.2 Outlook