A formal framework for optimizing and evaluating interactive retrieval system

Zhang, Yinan

Permalink

https://hdl.handle.net/2142/98262 Copy

Description

Title

A formal framework for optimizing and evaluating interactive retrieval system

Author(s)

Zhang, Yinan

Issue Date

2017-07-11

Director of Research (if dissertation) or Advisor (if thesis)

Zhai, ChengXiang

Doctoral Committee Chair(s)

Zhai, ChengXiang

Committee Member(s)

• Han, Jiawei
• Parameswaran, Aditya
• Fuhr, Norbert

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Information retrieval
• Probability ranking principle
• Interface card model

Abstract

The past decades have seen dramatic increase in the amount of information available to us, and the research area of information retrieval has served to help us access the tiny subset of information relevant to each of us more efficiently and more effectively. This thesis studies how to formally optimize as well as evaluate an interactive information retrieval system. First, we propose a formal general framework, the Interface Card Model, for optimizing interactive retrieval interface. We frame the task of an interactive retrieval system as to choose a sequence of interface cards to present to the user that can maximize the expected gain of relevant information for the user while minimizing the effort of the user, with consideration of the user's action model and any desired constraints on the interface card. We show that such a formal Interface Card Model can not only cover the classic Probability Ranking Principle as a special case by making multiple simplification assumptions, but also be used to derive a novel formal interface model for adaptively optimizing navigational interfaces in a retrieval system. Second, we propose a novel formulation of the Interface Card Model, the Interface Card Model with User States, for solving concrete interface optimization problems. The formulation is based on sequential decision theory, leading to a general framework for formal modeling of user states and stopping actions. Simulation and user study experiments demonstrate the effectiveness of the proposed model in automatically adjusting the interface layout in adaptation to inferred user stopping tendencies in addition to user interaction and screen size. Third, as a specific example of applying our proposed interface optimization framework in a larger scale real world application, we propose a Bayesian framework for user preference modeling and dynamically optimizing a faceted browsing system based on users' facet selection interactions. Finally, we propose a general formal framework for evaluating IR systems based on search session simulation that can be used to perform reproducible experiments for evaluating any IR system, including interactive systems and systems with sophisticated interfaces. We show that the traditional Cranfield evaluation method can be regarded as a special instantiation of the proposed framework where the simulated search session is a user sequentially browsing the presented search results. We further show that the proposed framework enables us to evaluate a set of tag-based search interfaces, a generalization of faceted browsing interfaces, producing results consistent with real user experiments and revealing interesting findings about effectiveness of the interfaces for different types of users.

Graduation Semester

2017-08

Type of Resource

text

Permalink

http://hdl.handle.net/2142/98262

Copyright and License Information

Copyright 2017 Yinan Zhang

Owning Collections