I. INTRODUCTION  
A. Related Surveys  
B. Survey Organization  

II. FUNDAMENTALS OF DL AND DISTRIBUTED DL  
A. Deep Learning  
B. Distributed DL Parallelism Modes  
C. Distributed DL Paradigms  

III. COMMUNICATION-EFFICIENT MODEL SYNCHRONIZATION  
A. Synchronous, Asynchronous, and Other Distributed SGD  
B. Convergence Guarantees of Distributed SGD  
C. Model Synchronization in FL  

IV. COMMUNICATION-EFFICIENT DATA COMPRESSION  
A. Gradient Quantization  
B. Gradient Sparsification  
C. Other Gradient Compression Technologies  

V. LARGE-SCALE RESOURCE ALLOCATION AND TASK SCHEDULING  
A. Resource Management  
B. Training-Task Scheduling  
C. Inference Scheduling  
D. Lessons Learned toward Large-scale Resource Allocation and Task Scheduling  

VI. LARGE-SCALE COMMUNICATION INFRASTRUCTURES  
A. GPU Interconnects  
B. Programmable Network Devices  
C. Inter-GPU Collective Communication  
D. Communication Topologies  
E. Lessons Learned toward High-Performance Large-scale Communication Infrastructures  

VII. LARGE-SCALE DISTRIBUTED TRAINING OF LARGE MODELS: A CASE STUDY ON LLMS  
A. Model Synchronization  
B. Communication Data Compression  
C. Resource Allocation and Task Scheduling  
D. Communication Infrastructure  
E. Large Foundation Models for Communications  

VIII. CONCLUSION