Regression tests - ProofDrivenQuerying/pdq GitHub Wiki

💠 Aim

The main aim of these tests is to verify the correctness of our software.

⚙ Setup

🗃 Data

Planner

18 examples available in our repository.
All the detailed information about them are available in our repository (in particular in the INFO.txt files available in the folders of each example).
Here are some key point:

  • They have different schemas, although some of them share the same schema
  • Only some of them contain data, so most of them cannot be run (they are designed only for planning)
  • The parameters used for each of them are available in the properties files of their respective folders
  • The "expected plans" represent the results of prior runs. When available, they can be found in their respective folder. They have an average of 24 operators with a deviation of ±14.

Runtime

2 examples available in our repository.
All the detailed information about them are available in our repository (in particular in the INFO.txt files available in the folders of each example).

📊 Results

These tests are automatically run nightly, and their results can be found here.

Planner

Here is a summary of the relevant part of one of the most recent logs at the time of writing (#421).

For each test we list the cost -- according to the cost function used as the default. We also list the set of accesses used. The tests mostly use an "approximate check" to see if the cost of the plan matches the cost of the previously found one.

Show the log 
Run export JARFILE=$(ls target/*-jar-with-dependencies.jar)
Creating log file: /home/runner/work/pdq/pdq/regression/TestResults/regression/2022-07-14T03:47:20.239.txt
Starting case '/home/runner/work/pdq/pdq/regression/test/planner/dag/MandatoryPass/SIMPLE_CONSTANT_rel_5x3_1_bindings+0.5_freeacc_dep_3x3_query_GUARDED_3' using java: 14.0.2

Running DAG GENERIC
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Using ApproximateCostAcceptanceCheck: PASS
	expected: 5.0 - [Access{V23,mt_23[]}]
	observed: 5.0 - [Access{V23,mt_23[]}]
  Duration: 0.14s.

Running DAG OPTIMISED
Using ApproximateCostAcceptanceCheck: PASS
	expected: 5.0 - [Access{V23,mt_23[]}]
	observed: 5.0 - [Access{V23,mt_23[]}]
  Duration: 0.02s.
Starting case '/home/runner/work/pdq/pdq/regression/test/planner/dag/MandatoryPass/SIMPLE_CONSTANT_rel_5x3_1_bindings+1.0_freeacc_dep_3x3_query_GUARDED_3' using java: 14.0.2

Running DAG GENERIC
Using ApproximateCostAcceptanceCheck: PASS
	expected: 5.0 - [Access{V39,mt_39[]}]
	observed: 5.0 - [Access{V39,mt_39[]}]
  Duration: 0.01s.

Running DAG OPTIMISED
Using ApproximateCostAcceptanceCheck: PASS
	expected: 5.0 - [Access{V39,mt_39[]}]
	observed: 5.0 - [Access{V39,mt_39[]}]
  Duration: 0.02s.
Starting case '/home/runner/work/pdq/pdq/regression/test/planner/dag/MandatoryPass/benchmark_case_002' using java: 14.0.2

Running DAG GENERIC
Using ApproximateCostAcceptanceCheck: PASS
	expected: 61394.5336050903 - [Access{part,m2[]}, Access{supplier,m20[]}, Access{partsupp,m6[#0=ps_partkey,#1=ps_suppkey]}]
	observed: 61394.5336050903 - [Access{part,m2[]}, Access{supplier,m20[]}, Access{partsupp,m6[#0=ps_partkey,#1=ps_suppkey]}]
  Duration: 2.21s.

Running DAG OPTIMISED
Using ApproximateCostAcceptanceCheck: PASS

Running DAG GENERIC
Using ApproximateCostAcceptanceCheck: PASS
	expected: 2.0 - [Access{Udirectory,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
	observed: 2.0 - [Access{Udirectory,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
  Duration: 0s.

Running DAG OPTIMISED
Using ApproximateCostAcceptanceCheck: PASS
	expected: 2.0 - [Access{Udirectory,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
	observed: 2.0 - [Access{Udirectory,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
  Duration: 0s.
Starting case '/home/runner/work/pdq/pdq/regression/test/planner/dag/MandatoryPass/profinfo/example_01b' using java: 14.0.2

Running DAG GENERIC
Using ApproximateCostAcceptanceCheck: PASS
	expected: 2.0 - [Access{Udirectory1,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
	observed: 2.0 - [Access{Udirectory1,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
  Duration: 0s.

Running DAG OPTIMISED
Using ApproximateCostAcceptanceCheck: PASS
	expected: 2.0 - [Access{Udirectory1,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
	observed: 2.0 - [Access{Udirectory1,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
  Duration: 0.01s.
Starting case '/home/runner/work/pdq/pdq/regression/test/planner/dag/MandatoryPass/profinfo/example_02' using java: 14.0.2

Running DAG GENERIC
Using ApproximateCostAcceptanceCheck: PASS
	expected: 4.0 - [Access{Ids,m4[]}, Access{Names,m6[]}, Access{Direct1,m3[#0=uname,#2=uid]}, Access{Direct2,m5[#0=uname,#1=addr]}]
	observed: 4.0 - [Access{Names,m6[]}, Access{Ids,m4[]}, Access{Direct1,m3[#0=uname,#2=uid]}, Access{Direct2,m5[#0=uname,#1=addr]}]
  Duration: 0.03s.

Running DAG OPTIMISED
Using ApproximateCostAcceptanceCheck: PASS
	expected: 4.0 - [Access{Ids,m4[]}, Access{Names,m6[]}, Access{Direct1,m3[#0=uname,#2=uid]}, Access{Direct2,m5[#0=uname,#1=addr]}]
	observed: 4.0 - [Access{Names,m6[]}, Access{Ids,m4[]}, Access{Direct1,m3[#0=uname,#2=uid]}, Access{Direct2,m5[#0=uname,#1=addr]}]
  Duration: 0.02s.
Starting case '/home/runner/work/pdq/pdq/regression/test/planner/dag/MandatoryPass/profinfo/example_03' using java: 14.0.2

Running DAG GENERIC
Using ApproximateCostAcceptanceCheck: PASS
	expected: 2.0 - [Access{Udirectory,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
	observed: 2.0 - [Access{Udirectory,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
  Duration: 0s.

Running DAG OPTIMISED
Using ApproximateCostAcceptanceCheck: PASS
	expected: 2.0 - [Access{Udirectory,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
	observed: 2.0 - [Access{Udirectory,m3[]}, Access{Profinfo,m2[#0=employeeid]}]
  Duration: 0.01s.
Starting case '/home/runner/work/pdq/pdq/regression/test/planner/dag/MandatoryPass/profinfo/example_03b' using java: 14.0.2

Running DAG GENERIC
Using ApproximateCostAcceptanceCheck: PASS
	expected: 2.0 - [Access{Udirectory,m3[#1=lastname]}, Access{Profinfo,m2[#0=employeeid,#1=lastname]}]
	observed: 2.0 - [Access{Udirectory,m3[#1=smith]}, Access{Profinfo,m2[#0=employeeid,#1=smith]}]
  Duration: 0s.

Running DAG OPTIMISED
Using ApproximateCostAcceptanceCheck: PASS
	expected: 2.0 - [Access{Udirectory,m3[#1=lastname]}, Access{Profinfo,m2[#0=employeeid,#1=lastname]}]
	observed: 2.0 - [Access{Udirectory,m3[#1=smith]}, Access{Profinfo,m2[#0=employeeid,#1=smith]}]
  Duration: 0.01s.

As can be seen, they all run in a few seconds.

Runtime

Here is a summary of the relevant part of one of the most recent logs at the time of writing (#421):

Show the log, with minor edits for readability 
Starting case '/home/runner/work/pdq/pdq/regression/test/runtime/DatabaseExamples/case_0[16](https://github.com/ProofDrivenQuerying/pdq/runs/7333215551?check_suite_focus=true#step:11:17)' using java: 14.0.2
Executing plan 658885585
plan 658885585 finished.
PASS
	Found 3 results as expected
Starting case '/home/runner/work/pdq/pdq/regression/test/runtime/MemoryExamples/case_001' using java: 14.0.2
Executing plan 1699493335
plan 1699493335 finished.
PASS
	Found 200000 results as expected
Starting case '/home/runner/work/pdq/pdq/regression/test/runtime/MemoryExamples/case_005' using java: 14.0.2
Executing plan 775763299
plan 775763299 finished.
PASS
	No result sets found
Starting case '/home/runner/work/pdq/pdq/regression/test/runtime/simple/rest_001' using java: 14.0.2
Executing plan 485741465
JerseyWebTarget { https://www.ebi.ac.uk/chembl/api/data/activity.json }
Received 20 amount of records.
plan 485741465 finished.
PASS
	Found 20 results as expected
Starting case '/home/runner/work/pdq/pdq/regression/test/runtime/simple/rest_002' using java: 14.0.2
Executing plan 9832[19](https://github.com/ProofDrivenQuerying/pdq/runs/7333215551?check_suite_focus=true#step:11:20)454
Calling web service JerseyWebTarget { http://pdq-webapp.cs.ox.ac.uk:8080/webapp/servlets/servlet/CountryFree }
Received 2 amount of records.
Calling web service JerseyWebTarget { http://pdq-webapp.cs.ox.ac.uk:8080/webapp/servlets/servlet/NationInput?n_nationkey=1 }
Received 1 amount of records.
Calling web service JerseyWebTarget { http://pdq-webapp.cs.ox.ac.uk:8080/webapp/servlets/servlet/NationInput?n_nationkey=2 }
Received 1 amount of records.
plan 983[21](https://github.com/ProofDrivenQuerying/pdq/runs/7333215551?check_suite_focus=true#step:11:22)9[45](https://github.com/ProofDrivenQuerying/pdq/runs/7333215551?check_suite_focus=true#step:11:46)4 finished.
PASS
	Found 2 results as expected

As can be seen, PDQ is able to retrieve the expected results.

PDQ - Proof-Driven Query Planning https://www.cs.ox.ac.uk/projects/pdq

Select the section you are interested in:

User Guides Development guidelines Developer Guides
⚠️ **GitHub.com Fallback** ⚠️