NEC Laboratories America

Tag Archive for: zaiyi zheng

You are here: / / zaiyi zheng

Zaiyi Zheng works at University of Virginia.

Posts

LOKA: Conflict-Aware LLM Knowledge Update with Adaptive Knowledge Memory

/in /by

Large Language Models (LLMs) have achieved remarkable success in natural language processing by encoding extensive knowledge, but their utility relies on timely updates as human knowledge keeps evolving. In this paper, we investigate the problem of LLM knowledge updates, which requires simultaneously unlearning un-wanted information and learning new knowledge. Existing approaches that tackle unlearning and learning separately encounter task conflicts and knowledge management issues when applied to comprehensive knowledge updates. In this paper, we validate our findings with theoretical analysis and empirical evidence, and propose LOKA, a conflict-aware framework for Large language mOdel Knowledge updAtes. During training, LOKA introduces an adaptive knowledge memory approach in which updated knowledge is allocated across multiple memory units. During inference, LOKA retrieves the most relevant memory unit from the knowledge memory and integrates it with the original LLM to apply updated knowledge, while a learning-based router controls the activation of the knowledge memory to improve knowledge utilization. Extensive experiments demonstrate the efficacy of LOKA in achieving accurate, flexible, and conflict-aware knowledge updates.

Read more

NEC Labs America Attends ACL 2026 San Diego July 2-7, 2026

/in /by

NEC Laboratories America heads to ACL 2026 in San Diego, California, July 2–7, to present accepted papers spanning knowledge updating and memory control in large language models, task-aware cultural alignment, uncertainty-aware reasoning, and adaptive chain-of-thought optimization, representing some of the most active frontiers in NLP and AI research today.

Read more

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion

/in /by

Symmetry in the parameter space of deep neural networks (DNNs) has proven beneficial for various deep learning applications. A well-known example is the permutation symmetry in Multi-Layer Perceptrons (MLPs), where permuting the rows of weight matrices in one layer and applying the inverse permutation to adjacent layers yields a functionally equivalent model. While permutation symmetry fully characterizes the equivalence set for MLPs, its discrete nature limits its utility for transformers. In this paper, we introduce rotation symmetry, a novel form of parameter space symmetry for transformers that generalizes permutation symmetry by rotating parameter matrices in self-attention layers. Unlike permutation symmetry, rotation symmetry operates in a continuous domain, thereby significantly expanding the equivalence set for transformers. Based on this property, we propose a theoretically optimal parameter matching algorithm as a plug-and-play module to enhance model fusion. We evaluate our approach using pre-trained transformers across diverse natural language and vision tasks. Experimental results demonstrate that our rotation symmetry based matching algorithm substantially improves model fusion, highlighting the potential of parameter space symmetry to facilitate model fusion. Our code is available on https://github.com/zhengzaiyi/RotationSymmetry.

Read more