[Colloq] Thesis Defense - Relevance Assessment (Un-)Reliability in Information Retrieval: Minimizing Negative Impact - Pavel Metrikov- Friday August 28th 12 pm WVH 366

[DiFazio, Danielle]

PhD Thesis Defense- Pavel Metrikov

Friday, August 28, at noon in 366 WVH

Title: Relevance Assessment (Un-)Reliability in Information Retrieval: Minimizing Negative Impact

Abstract:

Collecting relevance assessments is a very important procedure in Information Retrieval. It is conducted to (1) evaluate the performance of an existing search engine, or (2) build and train a new one. While most of the popular performance evaluation measures and search engine training algorithms assume the relevance assessments are accurate and reliable, in practice this assumption is often violated. Whether intentionally or not, assessors may provide noisy and inconsistent relevance judgments potentially leading to (1) incorrect conclusions about the performance of a search engine, or (2) inefficient or suboptimal training of a search engine.

Addressing the problem above, we first (a) demonstrate how one can quantify the negative effect of assessor disagreement (including intra-assessor disagreement as a special case) on the ranking performance of a search engine. Beside this theoretical result, we also propose practical recipes for (b) tuning existing evaluation measures with the goal of making them more robust to the label noise, (c) improving the reliability of relevance estimates by collecting and aggregating multiple assessments (potentially through crowdsourcing), and (d) incorporating a noise reduction component into learning-to-rank algorithms.

Committee:
- Jay Aslam (advisor)
- Mirek Riedewald
- David Smith
- Igor Kuralenok (external member)