Sequence-to-Sequence Neural Network
(Redirected from Neural sequence-to-sequence model)
Jump to navigation
Jump to search
A Sequence-to-Sequence Neural Network is a sequence-to-* model that is a neural network.
- Context:
- It can range from being a Simple Sequence-to-Sequence Neural Network to being a Complex Sequence-to-Sequence Neural Network.
- ...
- Example(s):
- Model Architecture-Specific Sequence-to-Sequence Neural Networks:
- RNN-based Neural Networks:
- Jordan's Recurrent Neural Network (1986), one of the earliest sequence-to-sequence models proposed by Michael I. Jordan, designed for producing temporal sequences.
- Elman's Simple Recurrent Network (SRN) (1990), developed by Jeffrey Elman, a sequence-to-sequence network that introduced the concept of context units for processing sequential data.
- LSTM-based Neural Networks:
- Google's Neural Machine Translation (GNMT), a specific implementation of a sequence-to-sequence network for language translation.
- Google's Magenta MusicVAE, a sequence-to-sequence network for music generation and interpolation.
- Transformer-based Neural Networks:
- OpenAI's GPT (Generative Pre-trained Transformer), a transformer-based sequence-to-sequence network used for various text generation tasks.
- Microsoft's CodeBERT, a sequence-to-sequence network pre-trained on programming language-natural language pairs for code-related tasks.
- Time Delay Neural Networks:
- Waibel's Time Delay Neural Network (TDNN) (1989), developed for phoneme recognition, one of the first sequence-to-class neural networks designed specifically for speech processing.
- ConvolutionalSequence-to-Sequence Neural Networks:
- Facebook's Wav2Vec, a sequence-to-sequence network designed for speech recognition tasks.
- Hybrid Architectures:
- DeepMind's AlphaFold, which can be considered a sequence-to-structure network for protein structure prediction, combining attention mechanisms with convolutional layers.
- OpenAI's DALL-E, a sequence-to-image network that generates images from text descriptions, using a transformer-like architecture with discrete VAE.
- Other Specialized Architectures:
- Bengio's Neural Probabilistic Language Model (2003), an early sequence-to-word model that learned distributed representations of words and laid the foundation for modern word embeddings.
- Sejnowski and Rosenberg's NETtalk (1987), a pioneering sequence-to-sequence network that converted written English text to phonemes for speech synthesis.
- RNN-based Neural Networks:
- Application-Specific Sequence-to-Sequence Neural Networks:
- text-to-text Neural Networks:
- Google's Neural Machine Translation (GNMT), a specific implementation for language translation, using LSTM-based architecture with attention mechanisms.
- OpenAI's GPT (Generative Pre-trained Transformer), a transformer-based network used for various text generation tasks, including translation, summarization, and question-answering.
- Bengio's Neural Probabilistic Language Model (2003), an early sequence-to-word model that learned distributed representations of words and laid the foundation for modern word embeddings.
- text/code-to-text/code Neural Networks:
- Microsoft's CodeBERT, a sequence-to-sequence network pre-trained on programming language-natural language pairs for code-related tasks such as code generation, documentation, and translation between programming languages.
- speech-to-text Neural Networks:
- Facebook's Wav2Vec, a self-supervised learning framework for speech recognition that uses convolutional neural networks to process raw audio waveforms.
- Waibel's Time Delay Neural Network (TDNN) (1989), developed for phoneme recognition, one of the first sequence-to-class neural networks designed specifically for speech processing.
- text-to-speech Neural Networks:
- Sejnowski and Rosenberg's NETtalk (1987), a pioneering sequence-to-sequence network that converted written English text to phonemes for speech synthesis.
- text-to-image Neural Networks:
- OpenAI's DALL-E, a sequence-to-image network that generates images from text descriptions, using a transformer-like architecture combined with a discrete variational autoencoder (VAE).
- sequence-to-structure Neural Networks:
- DeepMind's AlphaFold, a network for protein structure prediction that combines attention mechanisms with convolutional layers to predict 3D protein structures from amino acid sequences.
- text-to-music Neural Networks:
- Google's Magenta MusicVAE, a hierarchical recurrent variational autoencoder for music generation and interpolation, capable of producing long-term musical structures.
- general-purpose Sequence-to-Sequence Neural Networks:
- Jordan's Recurrent Neural Network (1986), one of the earliest sequence-to-sequence models proposed by Michael I. Jordan, designed for producing temporal sequences and applicable to various sequential tasks.
- Elman's Simple Recurrent Network (SRN) (1990), developed by Jeffrey Elman, introduced the concept of context units for processing sequential data, applicable to various sequence processing tasks.
- text-to-text Neural Networks:
- Model Architecture-Specific Sequence-to-Sequence Neural Networks:
- Counter-Example(s):
- ...
- See: Generative Neural Network.