Reflections on Tangible User Interfaces

Several of these posts, outlining the process of developing my first iPhone and Microsoft Surface applications, have been moved to my blog at http://lianapolitano.com. Want to learn from a beginner’s experimentations in data persistence and tangible user interface systems design? Check them out!

In “The Computer for the 21^st Century,” Mark Weiser describes the ubiquity of writing as a “background presence” of “literacy technology,” which does not require active attention but is prepared to deliver information at a glance. At this point in our history, computers are not similarly embedded in the world around us but are instead only present in “a single box.” Weiser calls for location-aware devices that are intelligently adaptive, and that are built to address specific tasks. He has a vision of “pads” which are to computers what scrap paper is to paper; useful anywhere and the “anitdote to windows.” The idea of mobile pads and live boards (all of the above involving displays) to create ubiquity is at the heart of Weiser’s vision for future computing.

Unlike Weiser, in “Tangible Bits: Towards Seamless Interfaces Between People, Bits and Atoms,” Ishii and Ullmer state their interest in moving away from GUIs nearly altogether, taking the idea of invisible computers quite literally and embedding them in everyday physical objects. As the authors themselves put it, they are more interested in “awakening richly-afforded physical objects, instruments, surfaces, and spaces to computational mediation, borrowing perhaps more from the physical forms of the pre-computer age than the present.” A key example is the ClearBoard, intended to change a wall “from a passive architectural partition to a dynamic collaboration medium.” The objects in the ambientROOM subtly display and communicate information by their very natures, and not by a concrete display of information on a GUI. As the authors put it, “GUIS fall short of embracing the richness of human senses and skills people have developed through a lifetime of interaction with the physical world.”

Our project is more within the scope of Weiser’s idea of ubiquitous computing, as it is an application written for a GUI, albeit a GUI embedded in a table surface. Particularly considering the project’s extension that I am developing on the iPhone, this system is comparable to pads (iPhone) and live boards (Surface), a bit more ubiquitous and spatially relevant to the different parts of the Davis Museum in which they will be used. Ways in which our project can be a bit more attuned to the goals of Ishii and Ullmer is by including iconography and objects in the application that are modeled on physical objects and interactions with them, moving away from common GUI idioms such as buttons to move forward and to close objects.

INTRODUCTION

The Token and Constraints (TAC) Paradigm described by Shaer, Leland, Calvillo-Gamez and Jacob in their paper The TAC Paradigm: Specifying Tangible User Interfaces includes several concepts and themes which illuminate certain Tangible User Interfaces. Based on Ullmer’s Token + Constraint approach to TUIs, the paradigm describes TUIs with “a set of relationships between physical objects and digital information…[where] a user may manipulate physical objects in order to access or manipulate digital information.” Its five key properties are Coupling, Relative Definition, Association, Computational Interpretation and Manipulation, which I apply in a case study of the MIT Media Lab’s Senseboard.

COUPLING

Coupling a physical token, or “pyfo,” with a variable creates a token by definition in the TAC paradigm. As an example in Senseboard, coupling information with a data “puck” creates a token. Using RFID technology, the data variable is associated with the given puck, and at run the data is projected onto its coupled puck.

RELATIVE DEFINITION

According to the researchers, this property declares that any “pyfo” may be defined as a token, a constraint or both. In Senseboard, command pucks and pucks without associated information are none of the above. Pucks that have been coupled to data are tokens, as defined above. Constraints in the Senseboard system include pucks that have already been placed, as they prevent other pucks from occupying that space, and also include any other structure controlling where the pucks can be placed on the board.

ASSOCIATION

An association describes the case when a token is physically associated with a constraint. This type of TAC relationship is exemplified by the association between a command puck and a given command in the Senseboard. When a command puck is being applied, not only does it impact the underlying puck but it temporarily obscures it, constraining its movement and preventing it from having other commands applied to it simultaneously.

COMPUTATIONAL INTERPRETATION

Manipulating a token within its constraints changes the state of the application via computational interpretation. Outside of the constraints, manipulation has no computational interpretation and, to the application, is irrelevant. In the case of Senseboard, when pucks are being maneuvered on the board it changes the state of the application, as the puck’s RFID sensors are interacting with the different RFID sensors across different spaces on the board. Similarly, applying command pucks on the board changes  the state of the pucks they are applied to, as their RFID sensor communicates with those of the puck below it and then the space on the Senseboard below that. However, when pucks (command or not) are maneuvered outside of the board (the main constraint of the system), the information and changes are not recorded and have no computational interpretation, as they are not communicated to the application.

MANIPULATION

This property describes the potential for each TAC to be manipulated discretely, continuously or in both ways. This manipulation is permitted within the application’s constraints and their physical properties. In Senseboard, pucks can be manipulated by adding, removing or moving the pucks across the board. Command pucks can be manipulated by placing them over the pucks on which they are to impact, and then removing them from said pucks. These discrete manipulations can be derived from the physical properties of the magnetized pucks and the encompassing size of the command pucks, which fit over the pucks they are meant to alter.

SUMMARY

INTRODUCTION

Senseboard, developed within the Tangible Media Group at the MIT Media Lab, is “a tangible interface platform for manipulating discrete pieces of abstract information.” The basic application developed on the platform is for scheduling conference talks, and the platform seeks to combine the benefits of physically manipulating note cards and of arranging information on a computer.

DESIGN CONCEPT

Information is represented both physically and digitally, by using magnetic “pucks” to represent blocks of time and projections onto the pucks to present their associated data. Commands are represented physically as specially shaped pucks, which can be overlaid onto the informational puck that is to be edited. These commands include copying/linking information, viewing details, grouping/ungrouping pucks, typing information, and exporting information. The shapes chosen to represent command pucks are tied into the function of that command, so that users can quickly grasp their meaning. The shapes chosen to represent informational pucks are designed to fit neatly into the Senseboard’s scheduling constraints.

IMPLEMENTATION

The main physical components of Senseboard are magnetized pucks, to be placed onto a large board. A projector causes the corresponding information for each puck to be projected onto the board based onto the puck’s placement. Additional pucks with a different physical look and size act as command pucks when placed over the informational pucks. The sensing technology used to implement the system is RFID technology, and it is used to track the position and associated information of each puck, so that the projector “knows” where to project given information.

USER EXPERIENCE

Typical users of the Senseboard are conference leaders and event planners seeking to organize their given conference or event. Seeking to collaboratively group presentations and organize them on a schedule is a typical use scenario for the Senseboard. The users interact with the Senseboard by moving pucks, updating their information and relationships with the command pucks, and exporting that information with the “Export” command puck. This TUI leverages the reality-based skills of allocating an activity to a “time slot,” and imposing a command upon an object (by accordingly placing a command puck over an informational puck). To update text, users need to bring in knowledge of how to use a keyboard from outside the application’s domain.

THE INTRODUCTION

Cell Tango, based on a concept developed by George LeGrady, is an interactive exhibit that creates its “art” via user-submissions and tagging on Flickr. Currently on display at Wellesley College’s Davis Museum, Wellesley students have been invited to submit their own photographs and tag-based descriptions to the Cell Tango Flickr account, so that the exhibit’s content is constantly expanding.

THE TECHNOLOGY

Using a projection linked to a computer, the exhibit’s content is projected onto a large screen in a darkened room. This technology only summarizes the method in which the content is output; the content is input through other devices (students’ computers and cell phones). The exhibit’s software applies algorithms to collect the most recently submitted photographs for display, collects related photographs on Flickr based on user tags, and populates the display with clusters based on tag relationships.

THE DESIGN

The tags that found these relationships are always stated explicitly on the screen, in any case where tags are used to group the photographs. This is the case in two screens, while in the other two tags are used either as accessories in a large mosaic, or are left out entirely as photographs tile over each other in layers. As a fan of the humane use of typefaces, it was crushingly sad to see that all of the tags were in a bright yellow Verdana. Aside from that, the display of photographs and tags was in a very minimalist style, with little to no graphic “effects,” display occurring on a solid black background, and bare lines used to draw connections in clusters.

THE ANALYSIS

Despite interplay between physical technologies for content creation, Cell Tango is hardly a TUI. The participants in this exercise are far removed from the exhibit itself, and have to physically attend the exhibit in order to see their contributions. Submitting material and witnessing its impact are two distinct activities, and this separation lessens the output’s relevance. Because input and output are removed rather than interwoven, the piece couldn’t be farther from Ishii’s definition of what a tangible user interface should be. Furthermore, the tangible input technologies used are not particularly tailored to the task at hand – by having to adhere to a sequence of interactions that could just as easily send a text message or write an e-mail, the tangible technology itself isn’t very intentional.

From an artistic vantage point, the piece falls short as well. Aesthetically, the work’s simplicity is less in the vein of Mondrian and more in the vein of an untrained web designer using Geocities. The aesthetic scheme seems largely unintentional, and reduces the “art” of the piece to the custom algorithms used. The project has not only an unattractive user interface, however, but the piece doesn’t even allow the viewer to analyze the work on their own, making it artistically shallow. By boldly incorporating the tags and making them the focus of the piece, they prevent the viewer from drawing their own conclusions about how clusters of photographs are related.

Because there is no room for interpretation and it looks like a primeval Powerpoint presentation to boot, this “piece” reads more as a screensaver than as an innovative technology for collaboration or an artistic effort worthy of renown. While it is a successful experiment in collecting data from external participants and mobile technology, its failure to make that data very expressive renders it superficial as an exhibit.