LUBM Benchmark
A LUBM Benchmark is a knowledge base system benchmark that ...
References
2009
- (Bizer & Schultz, 2009) ⇒ Christian Bizer, and Andreas Schultz. (2009). “The Berlin Sparql Benchmark.” In: International Journal on Semantic Web & Information Systems, 5(2).
- QUOTE: A benchmark is only a good tool for evaluating a system if the benchmark dataset and the workload are similar to the ones expected in the target use case (Gray, 1993; Yuanbo Guo et al, 2007). As Semantic Web technologies are used within a wide range of application scenarios, a variety of different benchmarks for Semantic Web technologies have been developed.
A widely used benchmark for comparing the performance, completeness and soundness of OWL reasoning engines is the Lehigh University Benchmark (LUBM) (Guo et al., 2005). In addition to the experiment in the original paper, (Rohloff et al., 2007) presents the results of benchmarking DAML DB, SwiftOWLIM, BigOWLIM and AllegroGraph using a LUMB (8000) dataset consisting of roughly one billion triples. The LUBM benchmark has been extended in (Ma et al., 2006) to the University Ontology Benchmark (UOBM) by adding axioms that make use of all OWL Lite and OWL DL constructs. As both benchmarks predate the SPARQL query language, they do not support benchmarking specific SPARQL features such as OPTIONAL filters or DESCRIBE and UNION operators. Both benchmarks do not employ benchmarking techniques such as system warm-up, simulating concurrent clients, and executing mixes of parameterized queries in order to test the caching strategy of a SUT.
- QUOTE: A benchmark is only a good tool for evaluating a system if the benchmark dataset and the workload are similar to the ones expected in the target use case (Gray, 1993; Yuanbo Guo et al, 2007). As Semantic Web technologies are used within a wide range of application scenarios, a variety of different benchmarks for Semantic Web technologies have been developed.
2006
- http://swat.cse.lehigh.edu/projects/lubm/
- Overview: The Lehigh University Benchmark is developed to facilitate the evaluation of Semantic Web repositories in a standard and systematic way. The benchmark is intended to evaluate the performance of those repositories with respect to extensional queries over a large data set that commits to a single realistic ontology. It consists of a university domain ontology, customizable and repeatable synthetic data, a set of test queries, and several performance metrics.
- Components:
- Ontology: The benchmark ontology is named Univ-Bench. It has two versions: OWL Version and DAML Version.
- Data Generator(UBA): This tool generates syntetic OWL or DAML+OIL data over the Univ-Bench ontology in the unit of a university. These data are repeatable and customizable, by allowing user to specify seed for random number generation, the number of universities, and the starting index of the universities.
- Test Queries: The benchmark currently provides 14 test queries. Here they are written in a KIF-like language, expressing a query as a conjunction of atoms. Examples of using other languages such as RQL can be found in the download of the test module below.
- Tester (UBT): The test module. It carries out both data loading test and query test, with configurable test plans.
2005
- (Guo et al., 2005) ⇒ Yuanbo Guo, Zhengxiang Pan, and Jeff Heflin. (2005). “LUBM: A Benchmark for OWL Knowledge Base Systems.” In: Web Semantics: Science, Services and Agents on the World Wide Web Journal, 3(2-3). doi:10.1016/j.websem.2005.06.005