[Colloq] Thesis Proposal Announcement - Maryam Bashir - Use and Utility of Crowd Sourced Preferences Judgements for Evaluation and Training - February 28th, 3:15pm, 366 WVH

[Jessica Biron]

PhD Thesis Proposal by: 

Maryam Bashir 

Date: Thu 28 Feb 2013 
Time: 3:15 pm 
Location: 366 WVH 


Title: Use and Utility of Crowd Sourced Preferences Judgements for Evaluation and Training 

Abstract: 
High quality relevance judgements are essential for the evaluation and training of information retrieval systems. Traditional methods of collecting relevance judgements are based on collecting binary or graded nominal judgements. However, nominal relevance judgements are limited by factors such as inter-assessor dis- agreement and the arbitrariness of grades. For this reason, we explore the use and utility of preference judgements in IR evaluation and training. Previous research has shown that it is easier for assessors to make pairwise preference judgments. However, unless the preferences collected are largely transitive, it is not clear how to combine them in order to obtain document relevance scores. Another difficulty is that the number of pairs that need to be assessed is quadratic in the number of documents. 

We propose to explore the problem of collecting preference judgements for the evaluation of IR systems through crowd sourcing, which requires that preference assessments from multiple human assessors be combined to ensure quality. Preliminary experiments in the collection and combination of preference judgements and our evaluation methodology demonstrate the potential of preference-based judgements for evaluation and training. 


Committee: 

Javed A. Aslam (Adviser) 
David A. Smith 
Yizhou Sun 
Mark D. Smucker (External Examiner)