2006 EvaluatingWordNetbasedMeasureso

(Budanitsky et al., 2006) ⇒ Alexander Budanitsky, and Graeme Hirst. (2006). “Evaluating WordNet-based Measures of Lexical Semantic Relatedness.” In: Computational Linguistics Journal, 32(1). doi:10.1162/coli.2006.32.1.13

Subject Headings: Lexical Similarity Measure Training.

Notes

Cited By

Quotes

Abstract

The quantiﬁcation of lexical semantic relatedness has many applications in NLP, and many different measures have been proposed. We evaluate ﬁve of these measures, all of which use WordNet as their central resource, by comparing their performance in detecting and correcting real-word spelling errors. An information-content–based measure proposed by Jiang and Conrath is found superior to those proposed by Hirst and St-Onge, Leacock and Chodorow, Lin, and Resnik. In addition, we explain why distributional similarity is not an adequate proxy for lexical semantic relatedness.

1. Introduction

The need to determine semantic relatedness or its inverse, semantic distance, between two lexically expressed concepts is a problem that pervades much of natural language processing. Measures of relatedness or distance are used in such applications as word sense disambiguation, determining the structure of texts, text summarization and annotation, information extraction and retrieval, automatic indexing, lexical selection, and the automatic correction of word errors in text. It’s important to note that semantic relatedness is a more general concept than similarity; similar entities are semantically related by virtue of their similarity (bank–trust company), but dissimilar entities may also be semantically related by lexical relationships such as meronymy (car–wheel) and antonymy (hot–cold), or just by any kind of functional relationship or frequent association (pencil–paper, penguin–Antarctica, rain–flood). Computational applications typically require relatedness rather than just similarity; for example, money and river are cues to the in-context meaning of bank that are just as good as trust company.

However, it is frequently unclear how to assess the relative merits of the many competing approaches that have been proposed for determining lexical semantic relatedness. Given a measure of relatedness, how can we tell whether it is a good one or a poor one? Given two measures, how can we tell whether one is better than the other, and under what conditions it is better? And what is it that makes some measures better than others? Our purpose in this paper is to compare the performance of a number of measures of semantic relatedness that have been proposed for use in applications in natural language processing and information retrieval.

1.1 Terminology and Notation

In the literature related to this topic, at least three different terms are used by different authors or sometimes interchangeably by the same authors: semantic relatedness, similarity, and semantic distance.

Resnik (1995) attempts to demonstrate the distinction between the first two by way of example. “Cars and gasoline”, he writes, “would seem to be more closely related than, say, cars and bicycles, but the latter pair are certainly more similar.” Similarity is thus a special case of semantic relatedness, and we adopt this perspective in this paper. Among other relationships that the notion of relatedness encompasses are the various kinds of meronymy, antonymy, functional association, and other “non-classical relations” (Morris and Hirst 2004).

The term semantic distance may cause even more confusion, as it can be used when talking about either just similarity or relatedness in general. Two concepts are “close” to one another if their similarity or their relatedness is high, and otherwise they are “distant”. Most of the time, these two uses are consistent with one another, but not always; antonymous concepts are dissimilar and hence distant in one sense, and yet are strongly related semantically and hence close in the other sense. We would thus have very much preferred to be able to adhere to the view of semantic distance as the inverse of semantic relatedness, not merely of similarity, in the present paper. Unfortunately, because of the sheer number of methods measuring similarity, as well as those measuring distance as the “opposite” of similarity, this would have made for an awkward presentation. Therefore, we have to ask the reader to rely on context when interpreting what exactly the expressions semantic distance, semantically distant, and semantically close mean in each particular case.

Various approaches presented below speak of concepts and words. As a means of acknowledging the polysemy of language, in this paper the term concept will refer to a particular sense of a given word. We want to be very clear that, throughout this paper, when we say that two words are “similar”, this is a short way of saying that they denote similar concepts; we are not talking about similarity of distributional or co-occurrence behavior of the words, for which the term word similarity has also been used (Dagan 2000; Dagan, Lee, and Pereira 1999). While similarity of denotation might be inferred from similarity of distributional or co-occurrence behavior (Dagan 2000; Weeds 2003), the two are distinct ideas. We return to the relationship between them in Section 6.2.

When we refer to hierarchies and networks of concepts, we will use both the terms link and edge to refer to the relationships between nodes; we prefer the former term when our view emphasizes the taxonomic aspect or the meaning of the network, and the latter when our view emphasizes algorithmic or graph-theoretic aspects. In running text, examples of concepts are typeset in sans-serif font, whereas examples of words are given in italics; in formulas, concepts and words will usually be denoted by c and w, with various subscripts. For the sake of uniformity of presentation, we have taken the liberty of altering the original notation accordingly in some other authors’ formulas.

References

,

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2006 EvaluatingWordNetbasedMeasureso	Graeme Hirst Alexander Budanitsky			Evaluating WordNet-based Measures of Lexical Semantic Relatedness				10.1162/coli.2006.32.1.13		2006