1 Introduction
2 Background: Transformers
2.1 Architecture of Transformers
2.2 Pretrained Transformers for Natural Language Processing
2.3 Pretrained Transformers for Computer Vision
3 Embedding Strategies
3.1 Textual Embeddings
3.2 Visual Embeddings
4 Model Architecture
4.1 Dual Encoders
4.2 Fusion Encoders
4.3 Combination Encoders
4.4 Encoder-Decoder Models
5 Pretraining Tasks
5.1 Masked Language Modeling
5.2 Masked Image Modeling
5.3 Image-Text Matching
5.4 Contrastive Learning
5.5 Visual Question Answering
5.6 Visual Grounding
5.7 Image Captioning
5.8 Prefix Language Modeling
5.9 Other Objectives
6 Downstream Capabilities
6.1 VL-Alignment
6.2 VL-Understanding
6.3 VL-Text Generation
6.4 Visual Grounding, Grounded Captioning and Object Detection
6.5 Image Generation
6.6 Video Tasks
6.7 Unimodal Tasks
7 Pretraining Data
7.1 Public Data Sources
7.2 Proprietary Datasets
7.3 Data Sizes
8 Analysis and Future Directions
8.1 Strengths
8.2 Limitations and Open Questions
8.3 Future Directions
8.4 Concluding Remarks