I. Introduction

II. Background

A. Language Models

B. Preference Learning

III. Feedback Sources

A. Direct Human Feedback

B. Feedback from Models

C. Feedback from Inductive Biases

IV. Feedback Formats

A. Relations Between Several Outputs

B. Properties Within Each Output

C. Combined Formats

V. Preference Modeling

A. Numerical Explicit Modeling

B. Natural Language Explicit Modeling

C. Implicit Modeling

VI. Preference Usage

A. Reinforcement Learning from Human Feedback

B. Supervised Fine-tuning on Preferred Outputs

C. Preference-guided Contrastive Learning

D. Preference-conditioned Fine-tuning and Generation

VII. Evaluation

A. Open-form Benchmarks

B. Automatic Evaluations

C. Qualitative Analyses

VIII. Conclusion and Outlooks