MaskGAN System

A MaskGAN System is an Automatic Text Generation System that can solve a MaskGAN Task.

• Context:
  • It was developed by Fedus|Fedus et al. (2018).
  • GitHub Repository: https://github.com/tensorflow/models/tree/master/research/maskgan
  • System's Architecture:
    • It is based on a MaskGAN Network which is composed by a Generator Network + a Discriminator Network+ a Critic Network.
    • It contains 2-layers of 650 LSTM units for both the generator and discriminator, 650 dimensional word embeddings, and variational dropout.
  • Training and other Tools:
    • It uses Reinforcement Learning and Policy Gradients Sutton et al. (1999) for estimating Generator Network gradients with respect to its parameters.
    • It includes a Pretrained language model using standard maximum likelihood training as in Luong et al. (2015).
    • It adopts a Adam method for stochastic optimization (Kingma & Ba, 2015) with default exponential decay rates $\beta_1$= 0.99 and $\beta_2=0.999$ for training baseline models;
    • It performs a Bayesian Optimization for hyperparameter tuning (variational droupout rate, learning rate) of generator, discriminator and critic networks;
• Example(s):
  • …
• Counter-Example(s):
  • a CJS Neural Narrative Text Generation Benchmark System,
  • a GSGAN Benchmark System,
  • a LeakGAN Benchmark System,
  • a LSPGS Wikipedia Long Sentences Summarization System,
  • MaliGAN Benchmark System,
  • RankGAN Benchmark System,
  • SeqGAN Benchmark System,
  • TextGAN Benchmark System,
  • Texygen Benchmark System.
• See: Neural Text Generation System, Seq2Seq Model, Neural Autoregressice Model, Professor Forcing Algorithm, Scheduled Sampling Algorithm.

---

---

References

2018

• (Fedus et al., 2018) ⇒ William Fedus, Ian Goodfellow, and Andrew M Dai. (2018). “MaskGAN: Better Text Generation via Filling in the ________". In: Proceedings of the Sixth International Conference on Learning Representations (ICLR-2018).

2015a

• (Kingma & Ba, 2015) ⇒ Diederik P. Kingma, and Jimmy Ba. (2015). “Adam: A Method for Stochastic Optimization.” In: Proceedings of the 3rd International Conference for Learning Representations (ICLR-2015).

2015b

• (Luong, Pham et al., 2015) ⇒ Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. (2015). “Effective Approaches to Attention-based Neural Machine Translation". In: Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP-2015). DOI:10.18653/v1/D15-1166.

1999

• (Sutton et al., 1999) ⇒ Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour (1999). "Policy Gradient Methods for Reinforcement Learning with Function Approximation". In: Advances in Neural Information Processing Systems 12 (NIPS Conference).

---