CONTENTS
1 INTRODUCTION
1.1 Literature Search and Selection
2 FOUNDATIONS
2.1 Embedding Layer
2.2 Transformer Encoder
2.3 Self-Supervised Learning
3 T-BPLMS CORE CONCEPTS
3.1 Pretraining Methods
3.2 Pretraining Tasks
3.3 Fine-Tuning Methods
3.4 Embeddings
4 T-BPLMS TAXONOMY
4.1 Pretraining Corpus
4.2 Extensions
5 BIOMEDICAL NLP TASKS
5.1 Natural Language Inference
5.2 Entity Extraction
5.3 Semantic Textual Similarity
5.4 Relation Extraction
5.5 Text Classification
5.6 Question Answering
5.7 Text Summarization
6 EVALUATION
7 CHALLENGES AND SOLUTIONS
7.1 Low Cost Domain Adaptation
7.2 Ontology Knowledge Injection
7.3 Small Datasets
7.4 Robustness to Noise
7.5 Quality In-Domain Word Representations
7.6 Low Resource (In-Domain Corpus) Pretraining
7.7 Quality Sequence Representation
8 FUTURE DIRECTIONS
8.1 Mitigating Bias
8.2 Privacy Issues
8.3 Domain Adaptation
8.4 Novel Pretraining Tasks
8.5 Benchmarks
8.6 Intrinsic Probes
8.7 Efficient Models
9 LIMITATIONS
10 CONCLUSION