In this preprint, the authors propose MEMIT, a method for updating language models with many memories, scaling up to thousands of associations for large models like GPT-J (6B) and GPT-NeoX (20B), exceeding prior work by orders of magnitude. The body of work on knowledge-editing methods is limited to updating at most a few dozen facts, whereas practical settings may require updating a model with hundreds or thousands of facts simultaneously. MEMIT modifies transformer weights to edit memories while maintaining generalization, specificity, and fluency at scales beyond other methods. The authors provide their code and data at memit.baulab.info.
