The test framework is used by quest-sesame and '''ontop-sparql-compliance''' modules to handle multi-variance cases that Quest must be able to handle. The framework is based on the source code in OpenRDF Data Access Working Group (DAWG) test cases. The framework offers a benefit in which developers can organize the test units in hierarchy thus allowing to group similar test cases.

We implemented this test framework in Ontop for the following test suites:

• Quest Completeness Test, located in package it.unibz.ontop.sesame.completeness.test in quest-sesame module.

• Quest Datatype Support Test, located in package it.unibz.krdb.obda.quest.datatypes in quest-sparql-compliance module.

• Quest Virtual Scenario Test, located in package it.unibz.krdb.obda.quest.scenarios in quest-sparql-compliance module.

• Quest SPARQL Compliance Test, located in package it.unibz.krdb.obda.quest.sparql in quest-sparql-compliance module.

A test suite consists of (1) Java code classes, (2) manifest files and (3) required input files.

In general, the Java code classes are arranged into 3 parts:

• The utility class. The class has a static method to aggregate the test suites coming from each test group and create a new complete test suite. It has also a static method to read the manifest file, parse the content as RDF triples and store them in a temporary triple storage.
• The parent class. It specifies the test unit: what is the test unit is about, how it is run, how the result is compared to the expected value and how the final results are displayed to users. The class is also responsible for collecting the test units and group them.
• The *-test class. It extends the parent class and it is the main entry class for executing the test cases. The JUnit TestRunner uses the static method suite() to run all known tests, i.e., all tests collected by the utility class.

There are two types of manifest file:

1. '''Main manifest file'''. This manifest file is used to link the location of the group tests and located in the testing root folder in src/test/resources folder. Below is a snippet from manifest-scenario-mysql.ttl file used by Quest Virtual Scenario Test. Notice that it lists the location of test groups and their corresponding manifest file.

<>  rdf:type mf:Manifest ;
   rdfs:label "Quest query evaluation scenario tests" ;
   mf:include (
       <bsbm/manifest-mysql.ttl>
       <fishmark/manifest-mysql.ttl>
       <iLog/manifest-mysql.ttl>
       <stockexchange/datatypes/manifest-mysql.ttl>
       <stockexchange/filters/manifest-mysql.ttl>
       <stockexchange/modifiers/manifest-mysql.ttl>
       <stockexchange/simplecq/manifest-mysql.ttl>
   ).

2. '''Test group manifest file'''. This manifest file is used to register and specify a test unit and located in the testing group folder. The snippet below shows an example taken from manifest-mysql.ttl file used by Quest Virtual Scenario Test. Notice the registration part mf:entries followed by the test unit specification.

<>  rdf:type mf:Manifest ;
   rdfs:label "BSBM Queries" ;
   mf:entries
   ( 
     :query-1
     :query-2
     :query-3
     :query-4
     ...
   ) .

:query-1 rdf:type mf:QueryEvaluationTest ;
   mf:name    "Q1: Find products for a given set of generic features" ;
   obdat:approval obdat:Approved ;
   obdat:approvedBy <http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/spec/20080912/index.html#queryTripleQ1> ;
   mf:action
           [ qt:query <query-1.rq> ] ;
   mf:result <query-result-1.ttl> ;
   mf:knowledgebase <bsbm.owl> ;
   mf:mappings <bsbm-mysql.obda> .

:query-2 rdf:type mf:QueryEvaluationTest ;
   mf:name    "Q2: Retrieve basic information about a specific product for display purposes" ;
   obdat:approval obdat:Approved ;
   obdat:approvedBy <http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/spec/20080912/index.html#queryTripleQ2> ;
   mf:action
           [ qt:query <query-2.rq> ] ;
   mf:result <query-result-2.ttl> ;
   mf:knowledgebase <bsbm.owl> ;
   mf:mappings <bsbm-mysql.obda> .
 
  ...

The test unit specification has several parameters that are mandatory, namely:

• mf:name : specifies the test name.
• mf:knowledgebase : specifies the location for the ontology file (in other test suite qt:ontology is used instead).
• mf:mappings : specifies the location for the mapping model file.
• qt:query : specifies the location for the query file
• mf:result : specifies the location for the expected result file.

Other optional parameters are:

• rdfs:comment : specifies the test comment.
• obdat:approval : specifies the status of the test: Approved or Rejected
• obdat:approvedBy : specifies the person or organization who approves it
• mf:parameters : specifies extra parameters to setup Quest

Below is a list of input files that the test suite is using:

1. OWL ontology file
2. OBDA mapping model file.
3. SPARQL query file.
4. Expected result file written in Turtle. The parameters for the expected result file is defined as followed:

• rsi:size : specifies the number of expected tuples returned from the query.
• rsi:thrownException : specifies the expected exception thrown by Quest. Note that if this parameter is used, set the rsi:size parameter as -1.

By default the logging level in Junit tests is set to "ERROR". This is done to minimize log file size, which is required for successful travis builds in Github commits. If one wishes, he could enable logging by setting logging level to "info" or "debug" locally. The properties file is located in src/test/resources/logback-test.xml in every project that has Junit tests.