1 INTRODUCTION  
1.1 Neural Network Design  
1.2 Neural Network Compression  
1.3 Neural Network Deployment  
1.4 Challenge and Future Work  
1.5 Contributions  

2 LIGHTWEIGHT ARCHITECTURE DESIGN  
2.1 Prior Knowledge of Lightweight Architecture  
2.2 Lightweight CNN Architecture  
2.3 Transformer-based Series  

3 FUNDAMENTAL METHODS IN MODEL COMPRESSION  
3.1 Pruning  
3.2 Quantization  
3.3 Knowledge Distillation (KD)  
3.4 Neural Architecture Search (NAS)  
3.5 Discussion and Summary  

4 HARDWARE ACCELERATION OF DEEP LEARNING MODELS  
4.1 Hardware Architectures  
4.2 Dataflow and the Data Locality Optimization  
4.3 Deep Learning Libraries  
4.4 Co-Design of Hardware Architecture  

5 CHALLENGE AND FUTURE WORK  
5.1 TinyML  
5.2 Building Lightweight Large Language Models  

6 CONCLUSION