On the Loss of Context Awareness in General Instruction Fine-tuning

Yihan Wang, Andrew Bai, Nanyun Peng, and Cho-Jui Hsieh, in Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS), 2025.

Download the full text

Abstract

Pre-trained Large Language Models (LLMs) require post-training methods such as supervised fine-tuning (SFT) on instruction-response pairs to enable instruction following. However, this process can cause forgetting in capabilities learned during pre-training. In this paper, we investigate the loss of context awareness after SFT, where context awareness is defined as the ability to extract and understand information from user-provided context. Surprisingly, we discovered that the loss of context awareness occurs in instruction fine-tuned LLMs when the chat template is applied to input prompts. We identify that the performance decline is associated with a bias toward different roles learned during conversational instruction fine-tuning. The bias can be traced to training samples where the assistant response minimally relies on the user-provided instruction. Based on these observations, we propose a metric to identify context-dependent examples from general instruction fine-tuning datasets. We then apply conditional instruction fine-tuning with a context-dependency indicator, enabling the model to preserve context awareness after SFT. Experiments on four context-dependent downstream tasks and three pre-trained LLMs of different sizes show that our method effectively mitigates the loss of context awareness without compromising general instruction-following capabilities.

Bib Entry

@inproceedings{wang2025context_awareness_loss,
  title = {On the Loss of Context Awareness in General Instruction Fine-tuning},
  author = {Wang, Yihan and Bai, Andrew and Peng, Nanyun and Hsieh, Cho-Jui},
  year = {2025},
  booktitle = {Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS)}
}