I. INTRODUCTION

II. KNOWLEDGE DISTILLATION
A. Distilled Knowledge and Loss
B. Teacher-Student Architecture
C. Distillation Process

III. KD APPLICATIONS
A. KD in Computer Vision (CV)
B. Natural Language Processing
C. Quantization

IV. DEEP LEARNING PARADIGM

V. RELATED WORKS
A. Learning Using Privileged Information (LUPI)
B. Generalized Distillation

VII. CONCLUSION