Investigating the Representation of Open Domain Dialogue Context for Transformer Models

Vishakh Padmakumar, Behnam Hedayatnia, Di Jin, Patrick Lange, Seokhwan Kim, Nanyun Peng, Yang Liu, and Dilek Hakkani-Tur, in Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2023.

Download the full text

Abstract

The bulk of work adapting transformer models to open-domain dialogue represents dialogue context as the concatenated set of turns in natural language. However, it is unclear if this is the best approach. In this work, we investigate this question by means of an empirical controlled experiment varying the dialogue context format from text-only formats (all recent utterances, summaries, selected utterances) as well as variants that are more structurally different (triples, AMR). We compare these formats based on fine-tuned model performance on two downstream tasks—knowledge selection and response generation. We find that simply concatenating the utterances works as a strong baseline in most cases, but is outperformed in longer contexts by a hybrid approach of combining a summary of the context with recent utterances. Through empirical analysis, our work highlights the need to examine the format of context representation and offers recommendations on adapting general-purpose language models to dialogue tasks.

Bib Entry

@inproceedings{padmakumar-etal-2023-investigating,
  title = {Investigating the Representation of Open Domain Dialogue Context for Transformer Models},
  author = {Padmakumar, Vishakh and Hedayatnia, Behnam and Jin, Di and Lange, Patrick and Kim, Seokhwan and Peng, Nanyun and Liu, Yang and Hakkani-Tur, Dilek},
  booktitle = {Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2023}
}