[Colloq] PhD Thesis Proposal by Virgil Pavlu - Monday, May 7

[Rachel Kalweit]

College of Computer and Information Science
presents
PhD Thesis Proposal by:
Virgil Pavlu

Proposal Title:
Large Scale IR Evaluation

Monday, May 7, 2007
1:30pm
366 West Village H

Abstract:
We consider the problem of large-scale retrieval evaluation, with a 
focus on the considerable effort required to accurately assess 
performance of retrieval systems using traditional techniques. It is 
clear now that this standard approach to evaluation of information 
systems by massively judging returned results is quickly becoming 
infeasible. We introduce two novel techniques for partial evaluation of 
retrieval systems together with empirical evidence of their effectiveness.

The first technique (HEDGE) presents a unified model which, given the 
ranked lists of documents returned by multiple retrieval systems in 
response to a given query, generates document collections likely to 
contain large fractions of relevant documents (pooling) and accurately 
evaluating the underlying retrieval systems with small numbers of 
relevance judgments (efficient system assessment); it also naturally 
provides a strategy for fusing the ranked lists of documents in order to 
obtain a high-quality combined list (metasearch). This approach is an 
adaptation of a popular on-line learning algorithm: in effect, our 
proposed system ``learns'' which documents are likely to be relevant 
from a sequence of on-line relevance judgments.

Our second technique (SAMPLING) randomly selects documents to be judged 
according to a given distribution.  The pool obtained is used for 
evaluation of retrieval systems.  While our estimates of performance are 
unbiased by statistical design, their variance is dependent on the 
sampling distribution employed; as such, we derive a sampling 
distribution likely to yield low variance estimates.  Our experiments 
indicate that highly accurate estimates of standard performance measures 
can be obtained using a number of relevance judgments as small as 4% of 
the typical judgment pools.

Thesis Committee:
Javed Aslam (thesis advisor)
Rajmohan Rajaraman
Ronald J Williams
Ian Soboroff (NIST)