Monte Carlo Methods for Managing Interactive State, Action and Feedback Under Uncertainty

Schwarz, J., Mankoff, J., Hudson, S., Monte Carlo Methods for Managing Interactive State, Action and Feedback Under Uncertainty. In Proceedings of 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, California, October 16-19, 2011). UIST ’11. ACM, New York, NY, 235 - 244.

paper video summary

This paper is the second of three research papers published as part of my PhD thesis.

One challenge I ran into when implementing the demos for my first paper was that I found it difficult to track that state of any user interface element that had more than two states. The aim of this paper was to modify the initial architecture to make it easy for developers to track the probabilistic state of user interface elements.

The key insight in this paper was to use a Monte Carlo approach to maintain a probabilistically accurate description of the user interface that can be used to make informed choices about actions. Samples are used to approximate the distribution of possible inputs, possible interactor states that result from inputs, and possible actions (callbacks and feedback) interactors may execute.

For example, a touch event is often represented as a single point, but in fact the finger is touching an entire area. My system approximates this distribution by generating a collection of individual touch events (Figure 2).

Figure 1. A single touch is modeled using a cloud of possible touch events. The system simulates what might happen to each possible event to decide what to do next.

Because each sample is certain, the developer can specify most of the behavior of interactors in a familiar, non-probabilistic fashion. This approach retains all the advantages of maintaining information about uncertainty while minimizing the need for the developer to work in probabilistic terms. This paper presents a working implementation of our framework and illustrates the power of these techniques within a paint program that includes three different kinds of uncertain input.

Probabilistic input events are sampled to produce a collection of possible input events. Each input event is dispatched and interface state is updated. Interface updated, creates a collection of possible actions. These actions are then resolved according to a configurable set of rules.

This video summary contains a demonstration of the capabilities of the framework as well as an explanation of the architecture.