1 Introduction  
2 Memory Wall  
2.1 Performance gap between device-based and server-based processors  
2.2 Models grow increasingly complicated  
2.3 Memory footprint gap between on-device training and inference  
3 Existing Approaches Analysis  
3.1 Hardware enable memory-friendly  
3.2 Model performance and memory trade-off  
3.3 Training efficiency and memory trade-off  
4 Future Directions  
5 Conclusion