[Colloq] PhD Thesis Defense - Virgil Pavlu, Monday, Aug. 18

[Rachel Kalweit]

Virgil Pavlu will be doing his PhD Thesis Defense on Monday, August 18 at 2:15pm in room 366 WVH.

Title: Large Scale IR Evaluation

Abstract:
We consider the problem of large-scale retrieval evaluation, with a focus on the considerable effort required to accurately assess
performance of retrieval systems using traditional techniques. It is clear now that this standard approach to evaluation of information systems by massively judging returned results is quickly becoming infeasible. We introduce two novel techniques based on random sampling for partial evaluation of retrieval systems together with empirical evidence of their effectiveness.
    
Our techniques  randomly select documents to be judged according to a given distribution. The pool obtained is used for evaluation of
retrieval systems. While our estimates of performance are unbiased by statistical design, their variance is dependent on the sampling
distribution employed; as such, we derive a sampling distribution likely to yield low variance estimates. Experiments indicate that highly accurate estimates of standard performance measures can be obtained using a number of relevance judgments as small as 4% of the typical judgment pools. Confidence intervals are computed based on estimates of variance; we discuss several interesting cases where the confidence serves as a warning of improper evaluation.

A massive IR experiment, Million Query Track 2007, used of one of the sampling techniques for evaluation. Out of 10000 topics used for testing IR systems, about 1800 topics have been judged for relevance, a first within the research community. We present the analysis of the track, conclusions and some future research directions.

Committee: 
Jay Aslam, Advisor
Ron Williams
Rajmohan Rajaraman
Ian Soboroff - NIST