[Colloq] REMINDER: PhD Thesis Proposal by Virgil Pavlu - TODAY, May 7

[Rachel Kalweit]

College of Computer and Information Science
presents
PhD Thesis Proposal by:
Virgil Pavlu

Proposal Title:
Large Scale IR Evaluation

Monday, May 7, 2007
1:30pm
366 West Village H

Abstract:
We consider the problem of large-scale retrieval evaluation, with a
focus on the considerable effort required to accurately assess
performance of retrieval systems using traditional techniques. It is
clear now that this standard approach to evaluation of information
systems by massively judging returned results is quickly becoming
infeasible. We introduce two novel techniques for partial evaluation of
retrieval systems together with empirical evidence of their effectiveness.

The first technique (HEDGE) presents a unified model which, given the
ranked lists of documents returned by multiple retrieval systems in
response to a given query, generates document collections likely to
contain large fractions of relevant documents (pooling) and accurately
evaluating the underlying retrieval systems with small numbers of
relevance judgments (efficient system assessment); it also naturally
provides a strategy for fusing the ranked lists of documents in order to
obtain a high-quality combined list (metasearch). This approach is an
adaptation of a popular on-line learning algorithm: in effect, our
proposed system ``learns'' which documents are likely to be relevant
from a sequence of on-line relevance judgments.

Our second technique (SAMPLING) randomly selects documents to be judged
according to a given distribution.  The pool obtained is used for
evaluation of retrieval systems.  While our estimates of performance are
unbiased by statistical design, their variance is dependent on the
sampling distribution employed; as such, we derive a sampling
distribution likely to yield low variance estimates.  Our experiments
indicate that highly accurate estimates of standard performance measures
can be obtained using a number of relevance judgments as small as 4% of
the typical judgment pools.

Thesis Committee:
Javed Aslam (thesis advisor)
Rajmohan Rajaraman
Ronald J Williams
Ian Soboroff (NIST)



_______________________________________________
Colloq mailing list
Colloq at lists.ccs.neu.edu
https://lists.ccs.neu.edu/bin/listinfo/colloq