Long Text Generation via Adversarial Training with Leaked Information (LeakGAN) System

A Long Text Generation via Adversarial Training with Leaked Information (LeakGAN) System is an automatic text generation system that implements an adversarial generative network training algorithm using leaked data to train a LeakGAN model.

• Context:
  • It's performance can be evaluated by a LeakGAN Benchmark Task.
  • a System's Architecture - LeakGAN Model.
  • Training System and other ML tools:
    • It uses a REINFORCE Algorithm (Williams 1992);
    • It uses bootstrapped rescaled activation, temperature control and interleaved training techniques.
  • It can be intended for Long Text Generation Tasks.
  • …
• Example(s):
  • https://github.com/CR-Gjx/LeakGAN, based on TensorFlow.
  • https://github.com/nurpeiis/LeakGAN-PyTorch, based on PyTorch.
  • …
• Counter-Example(s):
  • RankGAN System,
  • SeqGAN System,
  • SQuAD System,
  • Texygen System.
• See: Text Generation System, Natural Language Generation System, Natural Language Understanding System, Language Model, Reinforcement Learning Neural Network, FeUdal Network (FuN).

---

---

References

2018

• (Guo et al., 2018) ⇒ Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. (2018). “Long Text Generation via Adversarial Training with Leaked Information.” In: Proceedings of the Thirty-Second (AAAI) Conference on Artificial Intelligence (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th (AAAI) Symposium on Educational Advances in Artificial Intelligence (EAAI-18).
  • QUOTE: As illustrated in Figure 1, we specifically introduce a hierarchical generator $G$, which consists of a high-level MANAGER module and a low-level WORKER module. The MANAGER is a long short-term memory network (LSTM) (Hochreiter and Schmidhuber 1997) and serves as a mediator. In each step, it receives generator $D$’s high-level feature representation, e.g., the feature map of the CNN, and uses it to form the guiding goal for the WORKER module in that timestep. As the information from $D$ is internally-maintained and in an adversarial game it is not supposed to provide $G$ with such information. We thus call it a leakage of information from $D$.

Figure 1: An overview of our LeakGAN text generation framework. While the generator is responsible to generate the next word, the discriminator adversarially judges the generated sentence once it is complete. The chief novelty lies in that, unlike conventional adversarial training, during the process, the discriminator reveals its internal state (feature $f_t$) in order to guide the generator more informatively and frequently. (See Methodology Section for more details.)

1992

• (Williams, 1992) ⇒ Ronald J. Williams (1992). "Simple Statistical Gradient-following Algorithms for Connectionist Reinforcement Learning". In: Machine learning. DOI: https://doi.org/10.1007/BF00992696
  • QUOTE: This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units.

---