[Colloq] PhD Thesis Proposal by Emine Yilmaz, Monday, Nov. 13

[Rachel Kalweit]

College of Computer and Information Science
Presents

PhD Thesis Proposal by:
Emine Yilmaz
title:
Informative and Efficient Evaluation of Retrieval Systems

Monday, November 13, 2006
10:00am
366 West Village H

Abstract:
We consider the problem of evaluating retrieval systems. Despite the 
importance of evaluation in information retrieval, there are many 
questions that remain to be answered as to which evaluation measure is 
more "informative" and how one could efficiently compute the value of 
this measure.

In the first part of this proposal, we address three important questions 
regarding evaluation: Why are some evaluation measures considered as 
"better" than others? Among the most commonly used evaluation measures, 
which one is the most informative? Can the information captured by an 
evaluation measure be utilized to infer the relevance of documents?

In the second part of the proposal, given that we have defined the most 
informative evaluation measure among the most commonly used ones, we 
propose to investigate techniques for estimating the value of this 
measure using as few relevance judgments as possible. In order to do so, 
we first propose techniques based on sampling that assume that the 
incomplete judgments are a random subset of a complete judgment set. We 
further propose to investigate a method based on active learning that 
estimates the value of an evaluation measure using as few labeled data 
as possible by picking the most "informative" document to be judged.

Finally, we propose to show that given the estimates of average 
precision, one can infer the relevance of documents retrieved by that 
system. Combined, we propose to show how one can efficiently and 
accurately infer a large judged pool from a relatively small number of 
judged documents, thus permitting accurate and efficient evaluation on a 
large scale.

Committee:
Javed Aslam (advisor)
Harriet Fell
Ravi Sundaram
Stephen Robertson (Microsoft Research)