6. Evaluation of the system - AxelBremer/zoekmachines_week78 GitHub Wiki
Results
In order to evaluate the results of our search engine, 2 people have assessed the relevancy of the top 10 documents for 5 different queries independently. Relevant documents are denoted with 1 and irrelevant documents 0. The following 5 queries were evaluated:
- 'economy china increase', information need: What is known about the increasing economy of China and what consequences does the increasing economy have?
- 'mexico drugs war' , information need: What are the new developments on the topic of the Mexican drugs war?
- 'soviet nucleur weapons', information need: What was the Soviet Union doing with nuclear weapons?
- 'plane crash boeing', information need: What incidents occured where a boeing plane has crashed?
- 'Iran contra affair', information need: What was known about the Iran contra affair?
This resulted in the following relevance assesment table:
| Q/D | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| 1 | 1 1 | 1 1 | 1 1 | 1 1 | 1 1 |
| 2 | 1 1 | 1 1 | 1 1 | 1 1 | 1 1 |
| 3 | 1 1 | 0 0 | 1 1 | 0 0 | 1 1 |
| 4 | 1 0 | 0 0 | 1 1 | 1 1 | 1 0 |
| 5 | 1 1 | 0 0 | 1 1 | 1 1 | 1 1 |
| 6 | 0 0 | 0 0 | 1 1 | 1 0 | 1 1 |
| 7 | 1 1 | 0 0 | 1 1 | 0 0 | 1 1 |
| 8 | 1 1 | 0 0 | 0 0 | 0 0 | 1 1 |
| 9 | 0 0 | 0 0 | 1 1 | 0 0 | 1 1 |
| 10 | 1 0 | 0 0 | 0 0 | 1 1 | 1 1 |
Where the columns correspond to the query Q and the rows to to the nth document returned D. Each query-document pair is evaluated by two people therefore the first assesments are given by the evaluator 1 and the second by evaluator 2.
To get a better impression of the agreement between the two evaluators, the Cohen's Kappa is computed:
Cohen's kappa:
K = (po-pe)/(1-pe)
p(o) = 46/50
p(yes) = (34/50) * (30/50) = 0.408
p(no) = (16/50) * (20/50) = 0.128
pe = 0.408 + 0.128 = 0.536
K = 0.384/0.464 = 0.8275
So the cohen's kappa of the two judges is 0.83, which means that the observers are in relatively good agreement on the
assesments given to the documents.
Now we can calculate the average precision at 10 for our system based on the 5 queries, and the two relevance assesments. We have chosen to judge a retrieved as relevant when both evaluators assigned it a 1. This resulted in the following precision table for both evaluators together:
| Q | P |
|---|---|
| 1 | 6/10 |
| 2 | 2/10 |
| 3 | 8/10 |
| 4 | 5/10 |
| 5 | 9/10 |
If we add these precisions per query and divide this by 5 we get the following average precision: 6/10. This somewhat low accuracy can be explained by the fact that query 2 got remarkably bad assesments even though the first two documents where both judged relevant by both evaluators. This can be explained by the fact that the database we used simply does not contain more relevant articles on the Mexican drugs war. This one bad query has therefore caused the average precision to drop.
We also looked at the precision table of each evaluator separately, for evaluator 1 we obtained:
| Q | P |
|---|---|
| 1 | 8/10 |
| 2 | 2/10 |
| 3 | 8/10 |
| 4 | 6/10 |
| 5 | 1 |
For evaluator 2 we see:
| Q | P |
|---|---|
| 1 | 6/10 |
| 2 | 2/10 |
| 3 | 8/10 |
| 4 | 5/10 |
| 5 | 9/10 |
We immediately notice that the precision obtained for evaluator 2 are equal to the average precision of both assesments. This can easily be explained by the fact that there are no assesments in which evaluator 1 thought of a document as irrelevant while evaluator 2 judged it as relevant. To get a better visualization of the data we created the following graph:
From this graph we can see that query 2 and 3 were most easy to judge, whereas query 1, 4 and 5 were a bit harder.