TRECFiles Evaluator (for Windows)

TRECFiles Evaluator (ver 0.1 BETA) calculates the efficiency of the MAP, ANMRR, P10, P20 and the MNRO in TREC Files (Trec Format), giving also to the user the ability to execute significance tests (ST). STs tell us whether an observed effect, such as a difference between two means or a correlation between two variables, could reasonably occur just by chance in selecting a random sample. This application uses a bootstrap test, one-tailed, at significance levels 0.05, 0.01, and 0.001, against a baseline run.

The results of the developed application correlate with the results of TrecEval. TrecEval is the standard tool used by the TREC community for evaluating an ad hoc retrieval run, given the results and a standard set of judged results

Instructions:

1. Save your retrieval results in TREC format (i.e., just title, description, and narrative fields).

Example File

<query number> <1> <Rank> <Distance from the query image> <Label>

1 1 ucid00001 1 100 Rummager
1 1 ucid00226 2 80.4361997505131 Rummager
1 1 ucid00225 3 79.5920616609967 Rummager
1 1 ucid00490 4 77.8364116094987 Rummager
1 1 ucid00417 5 76.446932268008 Rummager
1 1 ucid00720 6 76.4177476124166 Rummager
1 1 ucid00202 7 76.1363636363636 Rummager
1 1 ucid01259 8 76.1179860840849 Rummager
1 1 ucid01266 9 75.2976605591834 Rummager

…..

Download Example File [CEDD performance on WANG database] [FCTH performance on WANG database]

2. Create the relevance assessments (qrels) file

<query number> <0> <Image from the Ground Truth> <1 (if is relevant)>

1 0 ucid00001 1
1 0 ucid00015 1
1 0 ucid00038 1
1 0 ucid00240 1
2 0 ucid00009 1
2 0 ucid00010 1
3 0 ucid00013 1
3 0 ucid00014 1
4 0 ucid00029 1

…..

Download the qrels for the [UCID database], and the [WANG database]

How to perform significance test?

Select a run as a baseline run. For example, you want to compare the performance of CEDD descriptor against the performance of FCTH descriptor.

1. Load the Qrels file
2. Load the CEDD’s Trec file
3. Load the FCTH’s Trec file as baseline
4. Press the Significance test button

For the latest version, visit http://cocytus.nonrelevant.net/thetrecfiles/

Questions and Suggestions: kzagoris@ee.duth.gr, schatzic@ee.duth.gr