LSPGS Wikipedia Long Sentences Summarization Task

A LSPGS Wikipedia Long Sentences Summarization Task is a Multi-Document Text Summarization Task that is based on a Transformer-based Neural Network architecture that generates Wikipedia articles.

• AKA: Text Summarization via Transformer Neural Network, Liu-Saleh-Pot-Goodrich-Sepassi Wikipedia Long Sentences Summarization Task, Liu's Wikipedia Long Sentences Summarization Task.
• Context:
  • Task Input(s): Wikipedia Topic.
  • Task Output(s): Wikipedia generated article.
  • Task Requirement(s):
    • Benchmark Datasets:
      • Enclish Gigaword dataset (Graff & Cieri, 2003),
      • CNN and DailyMail dataset (Nallapati et al., 2016),
      • WikiSum dataset (Liu et al., 2018)
    • Performance Metrics:
      • ROUGE scores.
      • Human evaluation ratings scores.
    • Baseline Models:
      • Extractive summarization models:
        • Identity Model-a trivial baseline extractor;
        • Term Frequency Inverse Document Frequency (TF-IDF) (Ramos, 2003);
        • TextRank (Mihalcea & Tarau, 2004);
        • Sumbasic (Nenkova & Vanderwende,2005);
        • Cheating Extractor Model;
      • Abstractive summarization models:
        • LSTM Encoder-Decoder with Attention (seq2seq-attn) (Bahdanau et al. 2014) ;
        • Transformer Neural Network (Vaswani et al., 2017) models:
          • Transformer Decoder (T-D);
          • Transformer Encoder-Decoder (T-ED);
          • Transformer Decoder with Memory-Compressed Attention (T-DMCA).
    • It can be solved LSPGS Wikipedia Long Sentences Summarization System that implements LSPGS Wikipedia Long Sentences Summarization Algorithms.
• Example(s):
  • Extractive methods comparison (Liu et al., 2018):

Extractor	Corpus	Test log-perplexity	ROUGE-L
cheating	combined	1.72975	59.3
tf-idf	combined	2.46645	34.2
tf-idf	citations-only	3.04299	22.6
tf-idf	search-only	3.56593	2.8
identity	combined	4.80215	4.0

• Models Performance (Liu et al., 2018):

  {{class="wikitable" style="border:1px; text-align:left; solid black; border-spacing:1px; margin: 1em auto; width: 80%"

|- ! Model !!Test perplexity!! ROUGE-L. |- |seq2seq-attention, $L = 500 || 5.04952|| 12.7 |- |Transformer-ED, $L = 500 || 2.46645|| 34.2 |- |Transformer-D, $L = 4000$|| 2.22216 ||33.6 |- | Transformer-DMCA, no MoE-layer, $L = 11000$ ||2.05159 ||36.2 |- |Transformer-DMCA, MoE-128, $L = 11000 || 1.92871|| 37.9 |- |Transformer-DMCA, MoE-256, $L = 7500$|| 1.90325|| 38.8 |- |}

• Counter-Example(s):
  • CJS Neural Narrative Text Generation Benchmark Task,
  • LKW Data-To-Text Generation Benchmarking Task,
  • Texygen Benchmark Task.
• See: Abstractive Text Summarization Task, Neural Abstractive Summarization Task, Self-Attention Mechanism, Sequence-to-Sequence (seq2seq) Neural Network, Natural Language Generation Task, Natural Language Understanding Task, Natural Language Translation Task, Natural Language Processing Task.

---

---

References

2018

• (Liu et al., 2018) ⇒ Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. (2018). “Generating Wikipedia by Summarizing Long Sequences.” In: Proceedings of the Sixth International Conference on Learning Representations (ICLR-2018).

2017

• (Vaswani et al., 2017) ⇒ Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. (2017). “Attention is all You Need.” In: Advances in Neural Information Processing Systems.

2016a

• (Nallapati et al., 2016) ⇒ Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos Santos, Caglar Gulcehre, and Bing Xiang. (2016). “Abstractive Text Summarization Using Sequence-to-sequence RNNs and Beyond.” In: Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL 2016). DOI:10.18653/v1/K16-1028.

2016b

• (Wu et al., 2016) ⇒ Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, and 27 co-authors. (2016). "Google's Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation". In: arXiv:1609.08144.

2015

• (Bahdanau et al., 2015) ⇒ Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. (2015). “Neural Machine Translation by Jointly Learning to Align and Translate.” In: Proceedings of the Third International Conference on Learning Representations, (ICLR-2015).

2005

• (Nenkova & Vanderwende, 2005) ⇒ Ani Nenkova, and Lucy Vanderwende (2005). “The Impact of Frequency on Summarization". Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005, 101.

2004

• (Mihalcea & Tarau, 2004) ⇒ Rada Mihalcea, and Paul Tarau. (2004). “TextRank: Bringing Order into Texts.” In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP 2004)

2003a

• (Graff & Cieri, 2003) ⇒ David Graff, and Christopher Cieri (2003). "English Gigaword". Linguistic Data Consortium, Philadelphia, 2003.
  • QUOTE: English Gigaword was produced by Linguistic Data Consortium (LDC) catalog number LDC2003T05 and ISBN 1-58563-260-0, and is distributed on DVD. This is a comprehensive archive of newswire text data in English that has been acquired over several years by the LDC.

    Four distinct international sources of English newswire are represented here:

    • Agence France Press English Service (afe).
    • Associated Press Worldstream English Service (apw).
    • The New York Times Newswire Service (nyt).
    • The Xinhua News Agency English Service (xie)

2003b

• (Ramos, 2003) ⇒ Juan Ramos (2003). “Using TF-IDF to Determine Word Relevance in Document Queries". In: Proceedings of the first instructional conference on Machine Learning, volume 242, pp. 133–142.

---