Sentence-BERT (SBERT) Algorithm

A Sentence-BERT (SBERT) Algorithm is a sentence embedding algorithm that modifies a pre-trained BERT model through the use of Siamese and triplet network structures.

• AKA: Sentence Embeddings using Siamese BERT-Networks (Bi-Encoder).
• Context:
  • It can (typically) significantly reduce the training time compared to previous Sentence Embedding Methods.
  • It can (typically) utilize pooling operations on the output of BERT/RoBERTa to derive fixed-sized sentence embeddings, experimenting with different strategies such as CLS-token output, mean of all output vectors (MEAN-strategy), and max-over-time of output vectors (MAX-strategy).
  • It can (often) be applied to various Natural Language Processing Tasks, including but not limited to semantic textual similarity, paraphrase identification, and information retrieval.
  • It can be implemented in SentenceTransformers package.
  • ...
• Example(s):
  • SBERT-NLI-base and SBERT-NLI-large are examples of SBERT models fine-tuned on the combination of SNLI and Multi-Genre NLI datasets for semantic textual similarity tasks, demonstrating superior performance over traditional sentence embedding methods.
  • one based on all-MiniLM-L6-v2 (~13.1MB), which maps text sentences & text paragraphs to a 384-dimensional dense vector space.
  • ...
• Counter-Example(s):
  • Universal Sentence Encoder, which uses deep averaging network (DAN) and transformer architecture to generate sentence embeddings.
  • InferSent, a model trained on natural language inference data to derive universal sentence representations.
  • BERT Embeddings without fine-tuning for sentence embeddings.
  • GloVe Embeddings used for generating sentence embeddings without the enhancements provided by SBERT's architecture.
  • OpenAI Embeddings.
• See: Sentence Embeddings, BERT, RoBERTa, Cosine Similarity, Neural Network Model, privateGPT, SentenceTransformers, Sentence Embedding.

---

---

References

2024

• https://www.sbert.net/
  • NOTE:
    • It utilizes Siamese and triplet network structures to derive semantically meaningful sentence embeddings, making it efficient for sentence-level tasks.
    • It significantly improves the performance of sentence embedding tasks by using BERT-based models in a Siamese network architecture for semantic similarity comparison.
    • It offers a method for reducing the computational cost and time required for embedding sentences, addressing the inefficiencies of directly applying BERT for sentence comparisons.
    • It enables more accurate and semantically meaningful comparisons between sentences than traditional methods, facilitating advancements in natural language understanding.
    • It has been evaluated extensively across various benchmarks, demonstrating state-of-the-art results in semantic textual similarity tasks.
    • It facilitates a wide range of applications, including semantic search, paraphrase identification, and text clustering, by providing highly informative sentence embeddings.
    • It encourages the NLP community to adopt more efficient and semantically aware models for sentence-level tasks, promoting research and development in this area.

2023

• https://towardsdatascience.com/sbert-deb3d4aef8a4
  • NOTE:
    • It introduces a Siamese network architecture to process sentences independently through the same BERT model, enabling efficient sentence embedding generation.
    • It applies a pooling layer after passing sentences through BERT to transform high-dimensional vectors into a single 768-dimensional vector, optimizing for dimensionality reduction.
    • It proposes three optimization objectives: classification, regression, and triplet, to cater to different NLP tasks and improve model performance across various applications.
    • It achieves massive improvements in processing speed by reducing the quadratic complexity of BERT inference to linear, significantly speeding up tasks like similarity search among large sentence collections.
    • It maintains high accuracy in embedding generation, demonstrating the effectiveness of its approach in capturing sentence semantics compared to traditional BERT embeddings.
    • It is fine-tuned on SNLI and MultiNLI datasets with a softmax classifier for classification tasks, indicating its robustness in understanding natural language inference.
    • It offers the SentenceTransformers library, providing a convenient interface for utilizing SBERT models for sentence embedding tasks, facilitating ease of use and accessibility for developers.

2019

• (Reimers & Gurevych, 2019) ⇒ Nils Reimers, and Iryna Gurevych. (2019). “[https://arxiv.org/pdf/1908.10084.pdf Sentence-BERT: Sentence Embeddings Using Siamese BERT-networks.” arXiv preprint arXiv:1908.10084
  • ABSTRACT: BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering.

    In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT.

    We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.

---