Title:

Bayesian decision theoretic approach to experimental design with application to usability experiments

This thesis looks at the practicality of applying a Bayesian Decision Theoretic
approach to the design of HCI usability experiments. It looks at the particular issues
involved in following the Bayesian experimental design framework of developing a
stochastic model, eliciting priors and utility functions and choosing the option with
the maximum expected utility.
HeI usability testing may involve user and analyst experimentation and various
courses of action may be employed using either one or both types of experiment.
The thesis shows that HeI usability experiments can be represented diagrammatically
by a decision tree so that courses of action and consequences can be shown in
sequential order and consequently that decision theory can be applied to experimental
design. A structure of three decisions is proposed for the user experiment where the
design of the experiment is a decision within the larger decision of whether to launch
or rewrite. A structure of a single decision is proposed for the analyst experiment.
The thesis shows that stochastic models can be developed which give solutions using
realistic priors and utility functions. For the user experiment the problem of a joint
prior distribution for two dependent binomial parameters is overcome by developing a
method using copula functions. For the analyst experiment a two factor capturerecapture
model for the identification of potential HeI problems is developed. Two
ways of representing the utility function, either in terms of monetary rewards only or
as a bivariate utility function, are investigated. The thesis shows that for realistic
utility functions both ways require numerical methods to calculate the expected
utilities, but a bivariate utility function has computational and elicitation advantages.
Hel usability experiments pose many questions including the following. Should a
user experiment be performed is it better to launch or rewrite without performing an
experiment? If a user experiment is performed what is the optimal number of
subjects? After a user experiment is it better to launch or rewrite? What is the optimal
number of analysts to take part in an experiment? How many problems are remaining
in the system after an analyst experiment? This thesis shows how models currently
described in the HCI literature can be generalised using a Bayesian Decision
Theoretic approach and used to give answers to these questions.
