I. Introduction  
II. Background on Transformer Networks  
A. Basic Modules  
B. Encoder and Decoder  
C. Family of Transformer Architectures  
D. Vision Transformer  
III. Motivation and Overview  
A. Motivation for optimizing transformer models  
B. Challenges for optimizing transformer models  
IV. Knowledge Distillation  
A. Overview of Knowledge Distillation Methods  
B. Methods based on task-awareness  
C. Methods based on distillation granularity  
V. Pruning  
A. Overview of pruning techniques  
B. Pruning taxonomy based on saliency quantification  
C. Classification based on the matrix sparsity pattern  
D. Classification based on Pruning granularity  
E. Quantitative comparison of pruning techniques  
F. Token/Patch Pruning  
G. Post Training Pruning (PTP)  
H. Hardware-aware pruning  
I. Storage formats for sparse matrices  
VI. Quantization  
A. Overview of quantization  
B. General quantization procedures  
C. Classification based on granularity of quantization  
D. Classification based on resultant bit-width or data-type  
E. Quantitative comparison of quantization techniques  
VII. Efficient Transformer Design  
A. Methods for NLP  
B. Methods for Computer Vision  
C. Quantitative comparison of lightweight CV transformer techniques  
VIII. Neural Architecture Search  
A. Overview of Neural Architecture Search  
B. Classification based on Transformer Search Space  
C. Classification based on Search Method  
D. Application of NAS for Model Compression  
E. Quantitative comparison of searched CV transformers  
IX. Hardware Optimization Techniques  
A. Pipelining  
B. Optimizing matrix-multiplication  
C. Skipping redundant, ineffectual or trivial computations  
D. Dataflows for exploiting reuse  
E. Block-circulant matrix for reducing weight storage  
X. Conclusion and Future Work