1 Introduction
2 Foundations in Tabular Data
2.1 Data Structures
2.2 Data Types
2.3 Downstream Tasks
2.4 Datasets
3 Language Modeling Techniques on Tabular Data
3.1 Input Processing
3.2 Intermediate Modules
3.3 Training Objectives
4 Evolution of How Language Models are Adapted
4.1 Pre-training From Scratch and PLMs
4.2 Emergence of Large Language Models
5 Challenges and Future Opportunities
5.1 Computation Efficiency
5.2 Interpretability
5.3 Biases
5.4 Data Types
6 Conclusion