[Colloq] Master's Thesis Defense - Applying EM to Compute Document Relevance from Crowdsourced Pair Preferences - Jie Wu - 4/26, 1pm, 366 WVH

[Jessica Biron]

Applying EM to Compute Document Relevance from Crowdsourced Pair Preferences 

Jie Wu 

Friday, April 26th 
1pm, 366 WVH 

Abstract: 

Traditionally, Information Retrieval (IR) systems evaluate based on absolute relevance judgments, obtained by hiring expert assessors. With the recent increase in the size of data corpora and the amount of queries, gathering labels from experts can be prohibitively expensive. One possible solution to this problem is to infer document relevance, where no true relevance labels are available, based on crowdsourced data. Experiments in this area have shown that noise in crowdsourced data is a significant obstacle to obtaining meaningful labels. We introduce an algorithm which uses Expectation Maximization (EM) and the Elo Rating System (Elo) on pairwise preference judgements to acquire document relevance from crowdsourced data. Our results show that by estimating the quality of crowd workers, EM reduces noise and generates accurate pair preference probabilities for IR system evaluation.