HKU and NVIDIA | factuality enhanced language models for open ended text generation

author ：Nayeon Lee , Wei Ping , Peng Xu , Mostofa Patwary , etc.

brief introduction ： For the pre training language model (LM) Open text generation ： This paper studies the improvement of large-scale LM The factual accuracy of . The author designed FactualityPrompts Test sets and metrics to measure LMs Authenticity . On this basis , The author studies the parameter size from 126M To 530B Of LMs The factual accuracy of . Interestingly , The author found that the larger LM The relatively small LM Is more in line with the facts , Although a previous study showed that , The larger LM Misunderstandings may not be true . Besides , A popular sampling algorithm in open text generation （ for example ,top-p） May be due to the introduction of “ Uniform randomness ” To the detriment of factuality . The author proposes a fact kernel sampling algorithm , The algorithm adapts to randomness dynamically to improve the authenticity of generation , While maintaining quality . Besides , The author analyzes the standard training method from the fact text corpus （ for example , Wikipedia ） Inefficiency in learning the correct association between entities in . The author puts forward a factual reinforcement training method , This method uses TopicPrefix Better understanding of facts and sentence completion as training objectives , Can greatly reduce factual errors .

 

Paper download ：https://arxiv.org/pdf/2206.04624.pdf