[Colloq] Thesis Proposal - Peter Golbus - Monday September 16th, 10:00am, 166 WVH

[Jessica Biron]

The College of Computer and Information Science Presents - Thesis Proposal Presentation: 

Speaker: Peter Golbus 

Date: Monday September 16, 2013 

Time: 10:00 AM 

Where: WVH 166 



Title: Beyond Measuring Performance: An Information-Theoretic Framework for the Evaluation and Analysis of Search Engines 



Abstract 



In order to improve the performance of search engines, it is necessary to accurately measure their performance. In this proposal, we demonstrate a probabilistic framework for using existing rank correlation measures to evaluate search engines. We show that interpreting this framework information-theoretically leads to powerful new tools for understanding search engines beyond simply measuring performance. 



In our preliminary work, we demonstrate the utility of our framework by developing (1) information tau, which can be used to measure conditional rank correlation, a powerful meta-evaluation tool whose use we demonstrate on understanding novelty and diversity evaluation, and (2) a new evaluation measure, relevance information correlation, which is correlated with traditional evaluation measures. This new measure can be used to (3) evaluate a collection of systems simultaneously, providing a natural upper bound on metasearch performance. 



We propose to explore the application of our framework to (1) rank correlation between preferences and partial rankings; (2) diversity evaluation based on diversity preferences; (3) a measure of the similarity between rankers on judged documents, information difference, which allows us to quantify the difference between systems with similar performance are in fact different; and (4) an application to web evaluation that can leverage different types of relevance assessments, e.g. relevance grades and pairwise document preferences, simultaneously. 



Thesis committee: 



Javed Aslam (advisor) 

David Smith 

Guevara Noubir 

Charles Clarke (University of Waterloo, external member) 





Jessica Biron 
Administrative Assistant – Co-op and Advising 
College of Computer and Information Science 
Northeastern University 
202 West Village H 
617-373-5204 
bironje at ccs.neu.edu 
http://www.ccs.neu.edu/