Controllable Text Generation with Neurally-Decomposed Oracle

Controllable Text Generation with Neurally-Decomposed Oracle

Tao Meng, Sidi Lu, Nanyun Peng, and Kai-Wei Chang, in Proceedings of the Thirty-Sixth Conference on Neural Information Processing Systems (NeurIPS), 2022.

Oral Paper (<2%)

Download the full text

Abstract

Bib Entry

@inproceedings{meng2022nado,
  title = {Controllable Text Generation with Neurally-Decomposed Oracle},
  author = {Meng, Tao and Lu, Sidi and Peng, Nanyun and Chang, Kai-Wei},
  booktitle = {Proceedings of the Thirty-Sixth Conference on Neural Information Processing Systems (NeurIPS)},
  year = {2022}
}

Related Publications

1. DiNADO: Norm-Disentangled Neurally-Decomposed Oracles for Controlling Language Models

   Sidi Lu, Wenbo Zhao, Chenyang Tao, Arpit Gupta, Shanchan Wu, Tagyoung Chung, and Nanyun Peng, in Proceedings of the Fortieth International Conference on Machine Learning (ICML), 2024.
   BibTeX Details

   @inproceedings{lu2024nado2,
     title = {DiNADO: Norm-Disentangled Neurally-Decomposed Oracles for Controlling Language Models},
     author = {Lu, Sidi and Zhao, Wenbo and Tao, Chenyang and Gupta, Arpit and Wu, Shanchan and Chung, Tagyoung and Peng, Nanyun},
     booktitle = {Proceedings of the Fortieth International Conference on Machine Learning (ICML)},
     year = {2024}
   }
   

   Details

2. Tractable Control for Autoregressive Language Generation

   Honghua Zhang, Meihua Dang, Nanyun Peng, and Guy Van den Broeck, in Proceedings of the Fortieth International Conference on Machine Learning (ICML), 2023.
   Full Text Abstract BibTeX Details Oral Paper (<2%)

   Despite the success of autoregressive large language models in text generation, it remains a major challenge to generate text that satisfies complex constraints: sampling from the conditional distribution [\Pr](\text[text] | α) is intractable for even the simplest lexical constraints α. To overcome this challenge, we propose to use tractable probabilistic models (TPMs) to impose lexical constraints in autoregressive text generation models, which we refer to as GeLaTo (Generating Language with Tractable Constraints). To demonstrate the effectiveness of this framework, we use distilled hidden Markov models, where we can efficiently compute [\Pr](\text[text] | α), to guide autoregressive generation from GPT2. GeLaTo achieves state-of-the-art performance on challenging benchmarks for constrained text generation (e.g., CommonGen), beating various strong baselines by a large margin. Our work not only opens up new avenues for controlling large language models but also motivates the development of more expressive TPMs.

   @inproceedings{zhang2023gelato,
     title = {Tractable Control for Autoregressive Language Generation},
     author = {Zhang, Honghua and Dang, Meihua and Peng, Nanyun and Broeck, Guy Van den},
     booktitle = {Proceedings of the Fortieth International Conference on Machine Learning (ICML)},
     year = {2023}
   }
   

   Details

3. Controllable Text Generation with Neurally-Decomposed Oracle

   Tao Meng, Sidi Lu, Nanyun Peng, and Kai-Wei Chang, in Proceedings of the Thirty-Sixth Conference on Neural Information Processing Systems (NeurIPS), 2022.
   Full Text BibTeX Details Oral Paper (<2%)

   @inproceedings{meng2022nado,
     title = {Controllable Text Generation with Neurally-Decomposed Oracle},
     author = {Meng, Tao and Lu, Sidi and Peng, Nanyun and Chang, Kai-Wei},
     booktitle = {Proceedings of the Thirty-Sixth Conference on Neural Information Processing Systems (NeurIPS)},
     year = {2022}
   }
   

   Details