ChaseBench benchmarks - ProofDrivenQuerying/pdq GitHub Wiki
💠 Aim
The main aim of these benchmarks is to analyze the performance of the new version of PDQ (v2.0). We see in particular that it performs better (i.e., passes more tests) than the previous one on ChaseBench.
⚙ Setup
A preliminary run of the ChaseBench benchmarks involved:
- Linux machine
- 32 cores
- 252 GB of RAM
- Fedora 5.10.8-200.fc33.x86_64
- PostgreSQL 14.3
- Our ChaseBench machinery
- A time limit of 300 minutes
- A memory limit of 30 GB
Please note that the configuration above is different from the one used in ChaseBench, since the machine used there is no longer available. So the specific values cannot be compared directly.
📊 Results
Legend:
- All values are expressed in seconds and are shown in the format:
EXTERNAL - INTERNAL. - If a value is missing (due to a failure, a timeout, or a memout), the symbol
~is shown. - Chase time: reasoning until termination time.
- Total time: chase time + query answering time.
| Deep | 100 | 200 | 300 |
|---|---|---|---|
| Chase time | 9 - 46 |
~ - 643 |
~ - ~ |
| Total time | 16 - 48 |
~ - ~ |
~ - ~ |
| Doctors | 10k | 100k | 500k | 1m |
|---|---|---|---|---|
| Chase time | 1 - 1 |
11 - 180 |
61 - ~ |
121 - ~ |
| Total time | 28 - 13 |
~ - ~ |
~ - ~ |
~ - ~ |
| DoctorsFD | 10k | 100k | 500k | 1m |
|---|---|---|---|---|
| Chase time | 1 - 1 |
6 - 5 |
27 - 92 |
52 - 341 |
| Total time | 3 - 2 |
14 - 15 |
55 - 138 |
106 - 460 |
| iBench | Ontology256 | STB128 |
|---|---|---|
| Chase time | 235 - 431 |
101 - 55 |
| Total time | 399 - ~ |
190 - ~ |
| LUBM | 1 | 10 | 100 | 1k |
|---|---|---|---|---|
| Chase time | 9 - 9 |
105 - 717 |
756 - ~ |
~ - ~ |
| Total time | 16 - 15 |
156 - ~ |
1269 - ~ |
~ - ~ |
Please note that these values are rounded off. They do suffice to determine when PDQ 2.0 can complete the tests or not.
As can be seen, at least one technique is able to complete the chase in a reasonable amount of time (less than 13 minutes), except for "Deep 300" and "LUBM 1k", and it is able to complete all the queries as well for most of them.
It is worth noting that the previous version of PDQ, as reported in the official ChaseBench results, was only able to complete one single benchmark ("Deep 100") within a time limit of 180 minutes.