Do LLMs Plan Like Human Writers? Comparing Journalist Coverage of Press Releases with LLMs

Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, and Mark Dredze, in Proceedings of The 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.

🏆 Outstanding Paper Award (<0.4%)

Abstract

Journalists engage in multiple steps in the news writing process that depend on human creativity, like exploring different “angles” (i.e., story directions). These can potentially be aided by large language models (LLMs). By affecting planning decisions, such interventions can have an outsize impact on creative output. We advocate a careful approach to evaluating these interventions, to ensure alignment with human values, by comparing LLM decisions to previous human decisions. In a case study of journalistic coverage of press releases, we assemble a large dataset of 250k press releases and 650k human-written articles covering them. We develop methods to identify news articles that challenge and contextualize press releases. Finally, we evaluate suggestions made by LLMs for these articles and compare these with decisions made by human journalists.

Bib Entry

@inproceedings{spangher2024llm_planning,
  author = {Spangher, Alexander and Peng, Nanyun and Gehrmann, Sebastian and Dredze, Mark},
  title = {Do LLMs Plan Like Human Writers? Comparing Journalist Coverage of Press Releases with LLMs},
  booktitle = {Proceedings of The 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2024}
}