Air+Touch

Chen, X., Schwarz, J., Harrison, C., Mankoff, J., and Hudson, S.E. Air+Touch: Interweaveing Touch & In-Air Gestures. In Proceedings of the 27th annual ACM symposium on User interface software and technology (Honolulu, Hawaii, October 5 - 9). UIST '14. ACM, New York, NY, USA, 519-525.

Air+Touch is a class of new interactions that interweave touch events with in-air gestures, offering a unified input modality with expressiveness greater than each input modality alone. This work demonstrates how air and touch are highly complementary: touch is used to designate targets and segment in-air gestures, while in-air gestures add expressivity to touch events.

For example, a user can draw a circle in the air and tap to trigger a context menu, do a finger 'high jump' between two touches to select a region of text, or drag and in-air ‘pigtail’ to copy text to the clipboard. To illustrate the potential of our approach, we built four applications that showcase seven exemplar Air+Touch interactions we created.

Building Air+Touch was a lot of fun; I led the technical implementation and project logistics (development of finger tracking algorithm & API using depth-sensing camera), while Anthony focused on demo implementation, design, and paper writing. One of my most positive experiences collaborating with another student.