Generative Adversarial Network (GAN) Model

A Generative Adversarial Network (GAN) is an Modular Neural Network that is composed by a Generator Neural Network and Discriminator Neural Network trained simultaneously by an GAN Training System.

• AKA: Generator-Discriminator Neural Network.
• Context:
  • It can be trained by a GAN Training System (that implements a GAN training algorithm).
  • It was initially developed by Goodfellow et al. (2014).
  • It can range from being a Bidirectional GAN (BiGAN) Model to being Deep Convolutional GAN (DCGAN) Model.
  • It can been trained to solve ML tasks from Image Generation Task to Text Generation Task.
  • …
• Example(s):
  • GAN models for Text Generation Task:
    • a Gumbel-Softmax GAN (GSGAN) Model (Kusner & Hernndez-Lobato, 2016),
    • a LeakGAN Model (Guo et al., 2018),
    • a MaliGAN Model (Che et al., 2017),
    • a MaskGAN Model (Fedus et al., 2018),
    • a RankGAN Model (Lin, Li, et al., 2017),
    • a SeqGAN Model (Yu, Zhang et al., 2017),
    • a TextGAN Model (Zhang et al., 2017),
  • GAN models for Image Generation Task:
    • a ImageNet GAN Model (Miyato & Koyama, 2018),
    • a Self-Attention GAN (SAGAN) Model (Zhang, Goodfellow et al., 2019).
  • …
• Counter-Example(s):
  • Gradient-Driven Diffusion Learning Model.
  • Encoder-Decoder Neural Network,
  • FeUdal Network,
  • Pointer-Generator Network,
  • Transformer-based Neural Network.
• See: Reinforcement Learning, Machine Learning, Ian Goodfellow, Neural Network, Zero-Sum Game, Generative Model, Unsupervised Learning, Semi-Supervised Learning, Supervised Learning.

---

---

References

2020a

• (Wikipedia, 2020) ⇒ https://en.wikipedia.org/wiki/Generative_adversarial_network Retrieved:2020-11-29.
  • A generative adversarial network (GAN) is a class of machine learning frameworks designed by Ian Goodfellow and his colleagues in 2014. Two neural networks contest with each other in a game (in the form of a zero-sum game, where one agent's gain is another agent's loss).

    Given a training set, this technique learns to generate new data with the same statistics as the training set. For example, a GAN trained on photographs can generate new photographs that look at least superficially authentic to human observers, having many realistic characteristics. Though originally proposed as a form of generative model for unsupervised learning, GANs have also proven useful for semi-supervised learning, fully supervised learning, and reinforcement learning. The core idea of a GAN is based on the "indirect" training through the discriminator, which itself is also being updated dynamically. This basically means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator. This enables the model to learn in an unsupervised manner.

2020b

• (TensorFlow, 2020) ⇒ https://www.tensorflow.org/tutorials/generative/dcgan Retrieved:2020-11-29.
  • QUOTE: Generative Adversarial Networks (GANs) are one of the most interesting ideas in computer science today. Two models are trained simultaneously by an adversarial process. A generator ("the artist") learns to create images that look real, while a discriminator ("the art critic") learns to tell real images apart from fakes.
    

    During training, the generator progressively becomes better at creating images that look real, while the discriminator becomes better at telling them apart. The process reaches equilibrium when the discriminator can no longer distinguish real images from fakes.

2019

• (Zhang, Goodfellow et al., 2019) ⇒ Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. (2018). “Self-Attention Generative Adversarial Networks.” “Self-attention Generative Adversarial Networks.” In: International Conference on Machine Learning (ICML-2019).
  • QUOTE: In this paper, we propose the Self-Attention Generative Adversarial Network (SAGAN) which allows attention-driven, long-range dependency modeling for image generation tasks. Traditional convolutional GANs generate high-resolution details as a function of only spatially local points in lower-resolution feature maps. In SAGAN, details can be generated using cues from all feature locations. Moreover, the discriminator can check that highly detailed features in distant portions of the image are consistent with each other. Furthermore, recent work has shown that generator conditioning affects GAN performance. Leveraging this insight, we apply spectral normalization to the GAN generator and find that this improves training dynamics.

2018a

• (Fedus et al., 2018) ⇒ William Fedus, Ian Goodfellow, and Andrew M Dai. (2018). “MaskGAN: Better Text Generation via Filling in the ________". In: Proceedings of the Sixth International Conference on Learning Representations (ICLR-2018).
  • QUOTE: We introduce an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context.
    (...)

    The discriminator has an identical architecture to the generator except that the output is a scalar probability at each time point, rather than a distribution over the vocabulary size. The discriminator is given the filled-in sequence from the generator, but importantly, it is given the original real context $\mathbf{m(x)}$. We give the discriminator the true context, otherwise, this algorithm has a critical failure mode.

    (...)

Figure 1: seq2seq generator architecture. Blue boxes represent known tokens and purple boxes are imputed tokens. We demonstrate a sampling operation via the dotted line. The encoder reads in a masked sequence, where masked tokens are denoted by an underscore, and then the decoder imputes the missing tokens by using the encoder hidden states. In this example, the generator should fill in the alphabetical ordering, (a, b, c, d, e).

2018b

• (Guo et al., 2018) ⇒ Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. (2018). “Long Text Generation via Adversarial Training with Leaked Information.” In: Proceedings of the Thirty-Second (AAAI) Conference on Artificial Intelligence (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th (AAAI) Symposium on Educational Advances in Artificial Intelligence (EAAI-18).
  • QUOTE: As illustrated in Figure 1, we specifically introduce a hierarchical generator $G$, which consists of a high-level MANAGER module and a low-level WORKER module. The MANAGER is a long short-term memory network (LSTM) (Hochreiter and Schmidhuber 1997) and serves as a mediator. In each step, it receives generator $D$’s high-level feature representation, e.g., the feature map of the CNN, and uses it to form the guiding goal for the WORKER module in that timestep. As the information from $D$ is internally-maintained and in an adversarial game it is not supposed to provide $G$ with such information. We thus call it a leakage of information from $D$.

Figure 1: An overview of our LeakGAN text generation framework. While the generator is responsible to generate the next word, the discriminator adversarially judges the generated sentence once it is complete. The chief novelty lies in that, unlike conventional adversarial training, during the process, the discriminator reveals its internal state (feature $f_t$) in order to guide the generator more informatively and frequently. (See Methodology Section for more details.)

2018c

• (Miyato & Koyama, 2018) ⇒ Takeru Miyato, and Masanori Koyama. (2018). “cGANs with Projection Discriminator.” In: Proceedings of the 6th International Conference on Learning Representations (ICLR 2018), Conference Track Proceedings.

2017a

• (Che et al., 2017) ⇒ Tong Che, Yanran Li, Ruixiang Zhang, R. Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. (2017). “Maximum-Likelihood Augmented Discrete Generative Adversarial Networks.” In: ArXiv Preprint: 1702.07983.

2017b

• (Lin et al., 2017) ⇒ Kevin Lin, Dianqi Li, Xiaodong He, Ming-ting Sun, and Zhengyou Zhang. (2017). “Adversarial Ranking for Language Generation.” In: Proceedings of Advances in Neural Information Processing Systems 30 (NIPS-2017).

2017c

• (Yu, Zhang et al., 2017) ⇒ Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. (2017). “SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient.” In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI 2017).

2017d

• (Zhang et al., 2017) ⇒ Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. (2017). “Adversarial Feature Matching for Text Generation". In: Proceedings of the 34th International Conference on Machine Learning (ICML 2017).

2016a

• (Goodfellow, 2016) ⇒ Ian Goodfellow (2016). "NIPS 2016 tutorial: Generative Adversarial Networks". In: arXiv:1701.00160.
  • QUOTE: The basic idea of GANs is to set up a game between two players. One of them is called the generator. The generator creates samples that are intended to come from the same distribution as the training data. The other player is the discriminator. The discriminator examines samples to determine whether they are real or fake. The discriminator learns using traditional supervised learning techniques, dividing inputs into two classes (real or fake). The generator is trained to fool the discriminator. We can think of the generator as being like a counterfeiter, trying to make fake money, and the discriminator as being like police, trying to allow legitimate money and catch counterfeit money. To succeed in this game, the counterfeiter must learn to make money that is indistinguishable from genuine money, and the generator network must learn to create samples that are drawn from the same distribution as the training data.

2016b

• (Kusner & Hernndez-Lobato, 2016) ⇒ Matt J. Kusner, and Jose Miguel Hernndez-Lobato. (2016). “GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution". In: arXiv:1611.04051.

2014

• (Goodfellow et al., 2014) ⇒ Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. (2014). “Generative Adversarial Nets.” In: Advances in Neural Information Processing Systems.

---