1. Introduction
2. Preliminaries
2.1. Introduction to Reinforcement Learning
2.2. Exploration
2.3. Challenging Problems
2.4. Benchmarks
3. Exploration in Reinforcement Learning
3.1. Prediction Error Methods
3.2. Reward Diverse Behaviours
3.3. Goal-Based Methods
3.4. Probabilistic Methods
3.5. Imitation-Based Methods
3.6. Safe Exploration
3.7. Exploration Parameters Methods
4. Future Challenges
5. Conclusions