Whereas large-scale neural language fashions, comparable to GPT2 and BART, have achieved spectacular outcomes on varied textual content era duties, they have a tendency to get caught in undesirable sentence-level loops with maximization-based decoding algorithms (e.g., grasping search). This phenomenon is counter-intuitive since there are few consecutive sentence-level repetitions within the human corpus (e.g., 0.02% in Wikitext-103). To research the underlying causes for producing consecutive sentence-level repetitions, we examine the connection between the likelihood of repetitive tokens and their earlier repetitions in context. Via our quantitative experiments, we discover that 1) Fashions have a desire to repeat the earlier sentence; 2) The sentence-level repetitions have a self-reinforcement impact: the extra instances a sentence is repeated within the context, the upper the likelihood of constant to generate that sentence; 3) The sentences with greater preliminary possibilities often have a stronger self-reinforcement impact. Motivated by our findings, we suggest a easy and efficient coaching technique DITTO (PseuDo-RepetITion PenalizaTiOn), the place the mannequin learns to penalize possibilities of sentence-level repetitions from artificial repetitive information. Though our technique is motivated by mitigating repetitions, our experiments present that DITTO not solely mitigates the repetition difficulty with out sacrificing perplexity, but additionally achieves higher era high quality. In depth experiments on open-ended textual content era (Wikitext-103) and textual content summarization (CNN/DailyMail) reveal the generality and effectiveness of our technique.
