CCS Concepts:  
• General and reference → Surveys and overviews  
• Computing methodologies → Artificial intelligence

1 INTRODUCTION

2 BACKGROUND

2.1 Vanilla Transformer  
2.2 Model Usage  
2.3 Model Analysis  
2.4 Comparing Transformer to Other Network Types  

3 TAXONOMY OF TRANSFORMERS

4 ATTENTION

5 OTHER MODULE-LEVEL MODIFICATIONS

6 ARCHITECTURE-LEVEL VARIANTS

7 PRE-TRAINED TRANSFORMERS

8 APPLICATIONS OF TRANSFORMER

9 CONCLUSION AND FUTURE DIRECTIONS