2023 EnhancingKnowledgeGraphConstruc
- (Trajanoska et al., 2023) ⇒ Milena Trajanoska, Riste Stojanov, and Dimitar Trajanov. (2023). “Enhancing Knowledge Graph Construction Using Large Language Models.” doi:10.48550/arXiv.2305.04676
Subject Headings: Automated KB Construction.
Notes
- The paper concludes that while both REBEL and ChatGPT could extract entities and relations from the articles and create knowledge bases, each model had its own strengths and weaknesses. REBEL was able to extract more standard relations and abstract concepts, while ChatGPT was able to identify a larger number of entities.
Cited By
Quotes
Abstract
The growing trend of Large Language Models (LLM) development has attracted significant attention, with models for various applications emerging consistently. However, the combined application of Large Language Models with semantic technologies for reasoning and inference is still a challenging task. This paper analyzes how the current advances in foundational LLM, like ChatGPT, can be compared with the specialized pretrained models, like REBEL, for joint entity and relation extraction. To evaluate this approach, we conducted several experiments using sustainability-related text as our use case. We created pipelines for the automatic creation of Knowledge Graphs from raw texts, and our findings indicate that using advanced LLM models can improve the accuracy of the process of creating these graphs from unstructured text. Furthermore, we explored the potential of automatic ontology creation using foundation LLM models, which resulted in even more relevant and accurate knowledge graphs.
Introduction
The technological advancements, together with the availability of Big Data, have led to a surge in the development of Large Language Models (LLMs) [1]. This trend has paved the way for a cascade of new models being released on a regular basis, each outperforming its predecessors. These models have started a revolution in the field with their capability to process massive amounts of unstructured text data and by achieving state-of-the-art results on multiple Natural Language Processing (NLP) tasks.
However, one of the aspects which have not yet taken over the spotlight is the combined application of these models with semantic technologies to enable reasoning and inference. This paper attempts to fill this gap by making a connection between the Deep Learning (DL) space and the semantic space, through the use of NLP for creating Knowledge Graphs [2].
Knowledge Graphs are structured representations of in-formation that capture the relationships between entities in a particular domain. They are used extensively in various applications, such as search engines, recommendation systems, and question-answering systems.
On a related note, there is a significant amount of raw texts available on the Web which contain valuable information. Nevertheless, this information is unusable if it cannot be extracted from the texts and applied for intelligent reasoning. This fact has motivated us to use some of the state-of-the-art models in an attempt to extract information from text data on the Web.
Yet, creating Knowledge Graphs from raw text data is a complex task that requires advanced NLP techniques such as Named Entity Recognition [3], Relation Extraction [4], and Semantic Parsing [5]. Large language models such as GPT-3 [6], T5 [7], and BERT [8] have shown remarkable performance in these tasks, and their use has resulted in significant improve- ments in the quality and accuracy of knowledge graphs.
To evaluate our approach in connecting both fields, we chose to analyze the specific use case of sustainability. Sustainability is a topic of great importance for our future, and a lot of emphasis has been placed on identifying ways to create more sustainable practices in organizations. Sustainability has become the norm for organizations in developed countries, mainly due to the rising awareness of their consumers and employees. However, this situation is not reflected in developing and underdeveloped countries to this extent. Although the perception of sustainability has improved, progress toward sustainable development has been slower, indicating the need for more concrete guidance [9]. Moreover, theoretical research has attempted to link strategic management and sustainable development in corporations in order to encourage the inte- gration of sustainability issues into corporate activities and strategies [10]. Even though research has set a basis for developing standards and policies in favor of sustainability, a more empirical approach is needed for policy definitions and analyzing an organization’s sustainability level with respect to the defined policies.
In this study, the goal is to make a connection between LLMs and semantic reasoning to automatically generate a Knowledge Graph on the topic of sustainability and populate it with concrete instances using news articles available on the Web. For this purpose, we create multiple experiments where we utilize popular NLP models, namely Relation Extraction By End-to-end Language generation (REBEL) [11] and Chat-GPT [12]. We show that although REBEL is specifically trained for relation extraction, ChatGPT, a conversational agent using a generative model, can streamline the process of automatically creating accurate Knowledge Graphs from an unstructured text when provided with detailed instructions.
The rest of the paper is structured as follows: Section II presents a brief literature overview, Section III describes the methods and experimental setup, Section IV outlines the results of the information extraction process, Section V states the propositions for future work, and finally section VIgives the conclusion of the work done in this paper.
II. LITERATURE REVIEW
A. Algorithms
Our study focuses on the task of information extraction from news and reports available on the Web. For this purpose, we compare the capabilities of NLP models to generate a useful Knowledge Base on the topic.
A Knowledge Base represents information stored in a structured format, ready to be used for analysis or inference. Often, Knowledge Bases are stored in the form of a graph and are then called Knowledge Graphs.
In order to create such a Knowledge Base, we need to extract information from the raw texts in a triplet format. An example of a triplet would be <Person, Location, City>. In the triplet, we have a structure consisting of the following links Entity -> Relation -> Entity, where the first entity is referred to as the subject, the relation is a predicate, and the second entity represents the object. In order to achieve this structured information extraction, we need to identify entities in the raw texts, as well as the relations connecting these entities.
In the past, this process was implemented by leveraging multi-step pipelines, where one step included Named-entity Recognition (NER) [3], and another step was Relation classi- fication (RC) [13]. However, these multi-step pipelines often prove to have unsatisfactory performance due to the propagation of errors from the steps. In order to tackle this problem, end-to-end approaches have been implemented, referred to as Relation-Extraction (RE) [4] methods.
One of the models utilized in this study is REBEL (Relation Extraction By End-to-end Language generation) [11], which is an auto-regressive seq2seq model based on BART [14] that performs end-to-end relation extraction for more than 200 different relation types. The model achieves 74 micro-F1 and 51 macro-F1 scores. It was created for the purpose of joint entity-relation extraction.
REBEL is a generative seq2seq model which attempts to ”translate” the raw text into a triple format. The REBEL model outputs additional tokens, which are used during its training to identify a triplet. These tokens include <triplet>, which represents the beginning of a triplet, <subj>, which represents the end of the subject and the start of the predicate, and <obj>, which represents the end of the predicate and start of the object. The authors of the paper for REBEL provide a parsing function for extracting the triplet from the output of REBEL.
The second approach we took was to use ChatGPT [12], as a conversational agent and compare the performance in the task of entity-relation extraction and creation of a common Knowledge Base. The agent consists of three steps, including separate models: a supervised fine-tuning (SFT) model based on GPT-3 [6], a reward model, and a reinforcement learning model.
ChatGPT was trained using Reinforcement Learning from Human Feedback (RLHF) [15], employing methods similar to InstructGPT with minor variations in data collection. An initial model is trained through supervised fine-tuning, with human AI trainers engaging in conversations, assuming both user and AI assistant roles. To aid in formulating responses, trainers were given access to model-generated suggestions. The newly created dialogue dataset was then combined with the InstructGPT dataset, which was transformed into a dialogue format. In order to establish a reward model for reinforcement learning, comparison data needed to be gathered, consisting of two or more model responses ranked by quality. This data was collected by taking conversations between AI trainers and the chatbot, randomly selecting a model-generated message, sampling multiple alternative completions, and having AI trainers rank them. The reward models# The provided text cuts off at the end.
...
References
;
| Author | volume | Date Value | title | type | journal | titleUrl | doi | note | year | |
|---|---|---|---|---|---|---|---|---|---|---|
| 2023 EnhancingKnowledgeGraphConstruc | Milena Trajanoska Riste Stojanov Dimitar Trajanov | Enhancing Knowledge Graph Construction Using Large Language Models | 10.48550/arXiv.2305.04676 Focus to learn more | 2023 |