Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM
Haw-Shiuan Chang, Nanyun Peng, Mohit Bansal, Anil Ramakrishna, and Tagyoung Chung, in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.
🏆 Best Paper Nomination (2%)
Abstract
Contrastive decoding (CD) improves the next-token distribution of a large expert language model (LM) using a small amateur LM. This paper theoretically explains why CD works well and introduces a new method, Asymptotic Probability Decoding (APD), to overcome its limitations. Experiments show that APD significantly boosts factuality in open-ended text generation and achieves new state-of-the-art results across multiple datasets.
Bib Entry
@inproceedings{chang2024contrastive,
title = {Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM},
author = {Chang, Haw-Shiuan and Peng, Nanyun and Bansal, Mohit and Ramakrishna, Anil and Chung, Tagyoung},
booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
year = {2024}
}