Your design is going to act in the world.

You need to understand your world in order to act appropriately. And you need to evaluate whether your design is acting as it ought to. Evaluation at all stages of the design process – from before you start the design to after you've deployed it – keeps your project on track and ultimately lowers the costs of mistakes. We are familiar with a number of evaluation methods appropriate for all stages of a design process. Each method presents certain advantages and disadvantages – we tailor our evaluation process to be appropriate to each particular design project.

Ethnography

Ethnographic field studies are a remarkably versatile evaluation method. Drawing on anthropological methods, ethnographic studies typically employ informal interviews, participant-observation, and careful qualitative data analysis. They can also be supplemented by the use of diaries, surveys, etc. Conducted at the beginning of a design process, an ethnographic study can supply a rich picture of existing practices that your design will address. A good ethnographic analysis can identify previously unanticipated pressure points that can be prodded via design. Conducted to assess a prototype or near-final design, we can generate a multifaceted understanding of the impact and appropriation of your design, in real-world contexts. Ethnographic studies typically require a greater time commitment than other evaluation methods, but the thoroughness of knowledge gained, often uncovering unanticipated opportunities, is well worth it.

Video Analysis

Taking video of people using, experiencing, or playing with an interactive system is a great way to understand what it's doing well and what it's doing badly. Close video analysis allows us to understand the minute details that contribute to user engagement. This is particularly effective at understanding how people get past an initial learning curve, and also excellent for showing how people collaborate while using your system. Video capture and analysis work best in stable spatial settings like galleries and offices, but is a bit more challenging for mobile systems.

Cultural Probes

Popularized in the field of interaction design by Bill Gaver at the Goldsmiths University of London, cultural probes are a design-led method emphasizing empathy and approachability, and participants' pleasure and active engagement. Probes given to participants are encountered like gift boxes, full of interesting objects like maps, stickers, postcards, and disposable cameras. Their intention is to elicit responses from people – about events, feelings and interactions in their everyday life – that can inspire innovative, pleasurable, evocative designs. This is a great method to use in the early phases of a creative project to inspire some truly original design ideas, but it is not suited to objective evaluation of near-complete designs.

User Testing

Generally performed in a controlled setting, like a lab or an office, a user test would involved certain well-defined tasks that one would perform with your system. This could involve a simple test of software usability, or a playtest to see how well rules, roles, and player enjoyment are balanced. Both qualitative and quantitative data can be gathered from such tests, for well-rounded evaluation. However, this method can easily fail to account for the social and spatial contexts of real use.

Paper Prototyping

You don't have to finish implementing a design in order to evaluate it! Paper prototypes concretize the appearance of your design and approximate some of its interactive properties. Testing them with users allows you to find problems and opportunities early in your design process, when they can still be acted on cheaply. This is a tried-and-true technique for graphical user interfaces, but with a little creativity, we've had great success testing paper prototypes of tangible and spatial projects as well. However, this method does not account for contexts of real-world use.

Focus Groups

Focus groups are a tried-and-true market research method. After identifying your target demographic(s), we recruit groups of people to interview together. The interview is usually semi-structured, including certain important questions, but providing plenty of room for participants to express their views. We can even do co-design exercises involving sketching, modeling clay, etc. Focus groups are a great way to identify your users' major desires very quickly, but can also miss design opportunities that participants forget to mention, or needs that they aren't consciously aware of.