In my work, I am often concerned with understanding people, problems they have, their context of product use, and the potential for technology and new ways of thinking to provide value to them. I use this understanding to create solutions that work. While my most valuable tools will always be creativity combined with pragmatism to shape different and better ideas into commercially viable solutions; one of the disciplines I have always found insightful is consumer research. Here I provide a short introduction to this topic.

Consumer research aims to capture, test and act on information about the behaviour of target consumers. It translates learnings from psychology, neuroscience & psychophysics into actionable insights for product design. It is most useful when developing products for people whose advantage is not clearly quantifiable with an existing technical performance metric (faster, stronger, or more productive). Consumer research can:

1. Identify the primary benefit of a solution and the secondary benefits this ladders to.

2. Identify the perceptual mechanisms by which this benefit is perceived.

3. Identify the variables that affect that perception and the associative links they have, both positive and negative

4. Create or select measures of perception for performance optimization.

The first two items might be different per market segment (groups of users with some similar behavioural characteristics easily identified by similar descriptive demographic characteristics) and so you should use this information to help select a target market, the latter might be different per product embodiment and can help compare and optimize products.

Data Collection Roadmap

Answering any research question requires stating that question clearly, gathering data and analysing it to provide the information on which to base a decision. A number of useful tools are available at every stage of the data collection roadmap to help you make a clear plan to do this.

1. Identify Questions & Hypothesis

   • 5 whys (root cause)

   • Identify beneficial effects

   • Cause & effect relationship

   • Measure size of effect

2. Identify Sources of Data

   • Process flow chart

   • Fishbone cause & effect (people, process, equipment, environment, materials, measurements)

   • CNX diagram (inputs, outputs, noise, constants)

3. Plan Data Collection & Validate Measurements

   • Measurement accuracy & repeatability

   • Formalised test methods

   • Sensory scales

4. Collect and Analyse Data

   • Normality test

   • Time series plot

   • Graphically stratified plots (multivariate)

   • Regression analysis

   • Statistical tests

5. Act on it!

   • Always understand why data is being collected and what it is for before starting data collection

Research Question with Null Hypothesis

The first step is to identify and ask a precise question, the answer to which will move the project forward in a useful way. A technique for doing this is to frame the question as a null hypothesis. A hypothesis (called H1) is a proposed explanation for a phenomena, often based on observations not satisfactorily explained by existing theory. Predictions of study outcomes can be tested before rejecting them. A null hypothesis (called H0) is the precise statement that there is no relationship between two measured phenomena (or two groups) that is used in formal science.

Consumer Research Methodologies

A means of answering that question can then be selected. The most common research methodologies that can be employed to answer a given question can be usefully grouped by their characteristics to better understand which question they might be most applicable to.

Error & Bias

Once a research question has been clearly expressed and a methodology selected, as that methodology is fully detailed, common sources of error and bias should be understood in order to ensure you are really answering the question you think you are.

Error is any difference between the values that were obtained from the sample of a study and the true values of the target population.

• Error of central tendency – participants tend to avoid extreme points on a scale

• Sample presentation order – the order in which samples are presented affects results

• Expectation error – consumer knowledge of a brand or product affects their perception

• Stimulus error – consumers with actual or perceived knowledge in an area can influence results

• Halo effect – one response can affect subsequent responses

• Contract error – an exaggeration or suppression of responses based on the selected sample set provided

Bias is any error that is systematic in nature.

• Researcher bias – researcher’s viewpoint will inevitably creep into question design and analysis. Double blind studies.

• Sample vs population – the sample is never a perfect match to the population you are trying to understand, asking the wrong people can skue results. Sampling methods are random, stratified, systematic, sub-group.

• Response bias – without a randomly selected sample, no statistical methods are valid. Surveys have a response rate of less than 10%. If the groups that respond differ from those that do not, this creates bias. Sampling with quotas (sub-group) or weighting data can compensate.

• Over correcting – don’t correct for an error or bias unless it has an effect on what you are asking.

Descriptive Statistics

Once data has been collected, descriptive statistics describe the basic features of a dataset in order to establish its validity. Before undertaking analysis, the distribution of the data set should be established using a histogram and simple scatter or time series plots can be used to find outliers and understand broad trends in your data.

• Averages – are a measure of a central tendency or typical value. Mean is the most common, calculated by summing the dataset and dividing by the number of values. It is heavily skewed by outliers and should be used on normally distributed data only. Median is the middle value and less skewed by outliers. Mode is the most frequent value(s) and is the only average suitable for discrete data and heavily skewed bimodal data.

• Margin of Error – the maximum expected difference between the true population parameter and a sample estimate of that parameter for a given confidence.

• Confidence in an Interval Estimate – confidence intervals specify a range within which the population truth is estimated to lie with a given probability. Calculations will vary depending on the distribution of the data.

• Range – the difference between smallest and largest values

• Interquartile range – the range of the middle 50% data points, gives a good indication of the amount of variation within the envelope of normal working practices

• Standard deviation – the average distance that each data point is from the mean (variance is standard deviation squared and useful mathematically). It is only valid for normal data

• Z-score – the number of standard deviations each data point is from the mean, Z-scores are unitless and so allow results of different parameters to be compared

• Standard error – estimated variation of the sample mean around the population mean

Inferential Statistics

Inferential statistics allow the use of data analysis to make inferences about a population. The type of data analysis to use should be selected on the basis of the datasets distribution, the type of data (Qualitative can nominal or ordinal, Quantitative can be concise or discreet) and what question is being asked:

• “Is there a pattern?” – Time series plot, scatter plot, plots by segment, multivariate analysis: Principal component analysis, clustering analysis, correspondence analysis (PCA for ordinal & nominal data)

• “Do people agree with each other?” – Inter-rater reliability (e.g. Kappa)

• “Which category is most important?” – Pareto chart

• “Why is there a difference?” – Logistic regression (binary, ordinal, nominal), liner regression

• “Is there a difference?” – See statistical significance

Statistical Significance

Achieving statistical significance is a statement that the results of a study were probably not due to chance. The probability (p-value) necessary to conclude that H0 is false is commonly required to be 95% (0.05), indicating there is only a 5% chance that the results of a study could have occured if H0 was false, leading to a type I error. Too high a p-value would increase the risk of a type II error. There are a number of other concepts that it is important to understand when running an experiment in which you intend to test for statistical significance.

• Outlier – an observation point that is distant from other observations, if due to experimental error, they should be excluded. There is no rigid mathematical definition for an outlier.

• Power – the probability that a test will correctly reject H0 (1 – β). Test power is commonly set at 80% (0.2, though this varies with statistical test type, and this allows the calculation of the sample size necessary for statistically significant results based on the magnitude of the effect of interest in the population. G*Power will perform this calculation. Effect size is generally estimated to be 0.5 before initial results have been received. It can be calculated as the mean of treatment group minus the mean of the control group and dividing it by the standard deviation of one of the groups.

• Sample Size – though this could be calculated through a power analysis once effect size is known, rules of thumb for continuous data can be given: Average value in population (10), Amount of variation in population (30), Frequency of values in categories (20), Relationships between variables (30), Stability over time (30). Discrete data depends on the size of the difference between points on the scale. Sample size should always be stated.

The statistical test to be used can be selected with this flow chart from Bates College.

Description of Methodologies

A wide number of methodologies are available for consumer research and, like any toolbox, it can be easy to forget what the ones you use least are good for. For this reasion I have collected a list of methodologies with brief descriptions below.

• Activity Analysis – List or represent in detail all tasks, actions, objects, performers and interactions involved in a process. This is a useful way to identify and prioritize which stakeholders to interview as well as which issues to address.

• Affinity Diagrams – Cluster design elements according to intuitive relationships such as similarity, dependence, proximity, etc. This method is a useful way to identify connections between issues and reveal innovation opportunities.

• Anthropometric Analysis – Use human population measurement data to check the coverage and suitability of the design solution for the target user group. This helps to identify a representative group of people for testing design concepts and evaluating the general usability of product details.

• Character Profiles – Based on observations of real people, develop character profiles to represent archetypes and the details of their behavior of lifestyles. This is a useful way to bring a typical consumer to life and to communicate the value of different concepts to various target groups.

• Cognitive Task Analysis – List and summaries all of the user’s sensory inputs, decision points and actions. This is good for understanding users perceptual, attentional and informational needs and to identify bottlenecks where errors may occur.

• Competitive Product Survey – Collect, compare, and conduct evaluations of the products competition. This is a useful way to establish functional requirements, performance standards and other benchmarks.

• Cross-Cultural Comparisons – Use personal or published accounts to reveal differences in behaviors and artefacts between national or other cultural groups. This helps teams to understand various cultural factors and the implications for their projects when designing for unfamiliar or global markets.

• Error Analysis – List all the things that can go wrong when using a product and determine the causes using a fishbone diagram. Material, measurement, machine, method, environment and people are categories. This is a good way to understand how design features mitigate or contribute to inevitable human errors and other failures. Input-output CNX diagrams for individual functions, listing controlled and uncontrolled variables can also be used.

• Five Whys? – Ask ‘Why?’ in response to 5 consecutive answers. This forces people to examine and express the underlying reasons for their behavior and attitudes.

• Flow Analysis – Represent the flow of information or activity through all phases of a system or process. This is useful for identifying bottlenecks and opportunities for functional alternatives.

• Historical Analysis – Compare features of an industry, organization, group, market segment or practice through various stages of development. This method helps to identify trends and cycles of product use and customer behavior and to project those patterns into the future.

• Long Range Forecasts – Write up prose scenarios that describe how social and/or technological trends might influence people’s behavior and the use of a product, service, or environment. Predicting changes in behavior, industry, or technology can help clients to understand the implications of design decisions.

• Secondary Research – Review published articles, papers, and other pertinent documents to develop an informed point of view on the design issues. This is a useful way to ground observations and to develop a point of view on the state of the art.

• Choice Based Conjoint – A questionnaire designed to determine what combination of a limited number of attributes is most influential on respondent choice or decision making. The relative value of the individual elements making up the product or service can be determined.

• Linguistic link Analysis – Link analysis identifies relationships, assigning strong links a high power. Text mining qualitative interview audio allows relationships between words that follow each other to be established.

• Regression analysis – find a pattern in quantitative data. Useful for predicting future data.

• Micro-expression – tracking micro-expressions can provide information on the subconscious system 1 reaction to certain stimulus.

• Eye Tracking – by tracking a user’s gaze, quantitative information on attention and hierarchy especially of websites and apps can be gathered.

• Reaction Time Testing – can be used to identify bias in subconscious system 1 heuristics. Useful for understanding predictable error in wider behavior.

• Minimum difference discrimination – tests the ability of the sensory system to discriminate between samples. This provides useful insight into minimum necessary tolerances.

• Case-Control Study – Most commonly carried out retrospectively, case control studies are used to compare cases who have a certain condition with a control group known not to have developed the outcome of interest. The control group is usually not only taken from the same population base, but also matched for age and gender.

• Market segmentation – Identify the people who make up the market for a product, describe them and gather attitudinal and behavioral information though a survey. Split this market into relevant sub-groups and describe each sub-group. This allows specific groups to be targeted, and key attitudinal information can feed into the design process.

• TRIZ Functional Analysis – Break a product down into basic functional expressions consisting of a subject, a verb and an object. Ensure the most basic, generalized words are used to allow for maximum number of solutions to be formulated. For instance, car = to transport people. Arrange these as a system & sub-system tree diagram. Further create function line diagrams, linking elements of a subsystem by their interactions. This helps to keep a design focused on a clear objective by pruning unnecessary functions and it ensure a clear, real problem is being addressed by the solution.

• DeBono’s 6 Hats – a discussion tool that assigns different team members specific roles for part of a brainstorm:

  • Managing Blue – what is the subject? what are we thinking about? what is the goal? Can look at the big picture.

  • Information White – considering purely what information is available, what are the facts?

  • Emotions Red – intuitive or instinctive gut reactions or statements of emotional feeling (but not any justification).

  • Discernment Black – logic applied to identifying reasons to be cautious and conservative. Practical, realistic.

  • Optimistic response Yellow – logic applied to identifying benefits, seeking harmony. Sees the brighter, sunny side of situations.

  • Creativity Green – statements of provocation and investigation, seeing where a thought goes. Thinks creatively, outside the box.

• Diffusion of Innovation Theory (Rodgers 1995) – Perceived characteristic of an invention explain its adoption rate by users/market. The characteristics are observability, relative advantage, compatibility, trial-ability, complexity.

• Kansei Engineering Software – The software follows the Kansei Engineering procedure suggested by Schütte (2006). {Additional info: Kansei Engineering is a method for translating feelings and impressions into product parameters.

• Interaction Relabelling – Interaction relabelling is a creativity method. In this method, participants are asked to consider an existing product, and, pretending that it is the product to be designed, to tell and act out how it works.

• A day in the life – Catalog the activities and contexts that users experience throughout an entire day. This is a useful way to reveal unanticipated issues inherent in the routines and circumstances people experience daily.

• Behavioral Archeology – Look for the evidence of people’s activities inherent in the placement, wear patterns, and organization of places and things. This reveals how artifacts and environments figure in people’s lives, highlighting aspects of their lifestyle, habits, priorities and values.

• Behavioral Mapping – Track the positions and movements of people within a space over time. Recording the pathways and traffic patterns of occupants of a space helps to define zones of different spatial behavior.

• Fly on the Wall – Observe and record behavior within its context, without interfering with people’s activities. It is useful to see what people actually do within real contexts and time frames, rather than accept what they say they did after the fact.

• Guided Tours – Accompany participants on a guided tour of the project-relevant spaces and activities they experience. Making an exploration of objects and actions in situ helps people recall their intentions and values.

• Personal Inventory – Document the things that people identify as important to them as a way of cataloging evidence of their lifestyles. This method is useful for revealing people’s activities, perceptions and values as well as patterns among them.

• Rapid Ethnography – Spend as much time as you can with people relevant to the design topic. Establish their trust in order to visit and participate in their natural habitat and witness specific activities. This is a good way to achieve a deep first hand understanding of habits, rituals, natural language and meanings around relevant activities and artifacts.

• Shadowing – tag along with people to observe and understand their day-to-day routines, interactions and contexts. This is a valuable way to reveal design opportunities and show how a product might affect or complement user’s behavior.

• Social Network Mapping – Notice different kinds of social relationships within a user group and map the network of their interactions. This is a useful way to understand interpersonal and professional relationship structures within a workgroup.

• Sill Photo Survey – Follow a planned shooting script and capture pictures of specific objects, activities. The team can use this visual evidence to uncover patterns of behavior and perceptions related to a particular product or context, as well as structure and inspire design ideas.

• Time-Lapse Video – Setup a time-lapse camera to record movements in a space over an extended period of time. Useful for providing an objective, longitudinal view of activity within a context.

• Ethnography – Rather than asking a user what they do, watch them. As what people say is often different from what they do.

• Camera Journal – Ask potential users to keep a written and visual diary of their impressions, circumstances and activities related to the product. This rich, self-conducted notation technique is useful for prompting users to reveal points of view and patterns of behavior.

• Card Sort – On separate cards, name possible features, functions or design attributes. Ask people to organize the cards spatially, in ways that make sense to them. This helps to expose peoples metal models of a device or system. Their organization reveals expectations and priorities about the intended functions.

• Cognitive Maps – Ask participants to map an existing or virtual space and show how they navigate it. This is a useful way to discover the significant elements, pathways and spatial behavior associated with an environment.

• Collage – Ask participants to build a collage from a provided collection of images and explain the significance of the images and arrangement they chose. This illustrates participants understanding and perceptions of issues and helps them verbalize complex or unimagined themes.

• Conceptual Landscape – Diagram, sketch or map the aspects of abstract social and behavioral constructs or phenomena. This is a helpful way to understand people’s mental models of the issues related to the design problem.

• Cultural Probes – Assemble a camera journal kit and distribute it to participants within one or across many cultures to collect and evaluate perceptions and behavior within or across cultures.

• Draw the Experience – Ask participants to visualize an experience through drawings and diagrams. This can be a good way to debunk assumptions and reveal how people conceived of and order their experiences and activities.

• Extreme User Interviews – Identify individuals who are extremely familiar or completely unfamiliar with the product and ask them to evaluate their experience using it. These individuals are often able to highlight key issues of the design problem and provide insights for design improvements.

• Foreign Correspondents – Request input from coworkers and contacts in other countries and conduct a cross-cultural study to derive basic international design principles. This is a good way to illustrate the cultural and environmental contexts in which products are used.

• Narration – As they perform a process or execute a specific task, ask participants to describe aloud what they are thinking. This is a useful way to reach users motivations, concerns, perceptions and reasoning.

• Surveys & Questionnaires – Ask a series of targeted questions in order to ascertain particular characteristics and perceptions of users. This is a quick way to elicit answers from a large number of people.

• Un-focus Groups – Assemble a diverse group of individuals in a workshop to use a stimulating range of materials and create things that are relevant to your project. Encourages rich, creative and divergent contributions from potential users, releases inhibitions and opens up new thinking.

• Word-Concept Association – Ask people to associate descriptive words with different design concepts of features to show how they perceive and value the issues. Clustering users’ perceptions helps to evaluate and prioritize design features and concepts.

• Sentence completion – After using a system, a participant is handed a set of beginnings of sentences that she then completes. The beginnings of the sentences trigger the user think the experiential aspects of product use, e.g.”When I use this product, I feel myself…”, or “The appearance of this product is…”

• Focus Groups – Assemble similar groups and guide discussion with a moderator to tease out in depth qualitative information on their attitudes and behaviors.

• Product concept acceptance – Fully describe a product concept to a group of users and ask for their thoughts, if they like it, ask them if they would like to buy it in order to establish acceptance for the proposal. If not, how would they change it?

• Quantitative descriptive analysis – uses a trained descriptive panel to measure a product’s sensory characteristics. Panel members use their senses to identify perceived similarities and differences in products, and articulate those perceptions in their own words.

• Sensory Testing – Rate people’s response to stimulus on based on the perception from different senses. This allows you to understand what physical characteristics drive what perceptual impressions and overall user preference.

• A/B Trial – Two concepts are rated in comparison to each other by two user groups in order to make clear development decisions.

• Max. Difference/best worst ranking/paired preference – A maximum difference questionnaire compares 40 options against each other in every combination, establishing a clear hierarchy of user preference.

• Emotional Rating – Use an emotional rating scale to establish response to stimulus. The affect grid / Geneva Emotion Wheel uses a 2D space with the axis activating/deactivating and positive/negative. Other scales include Likert 7 point scale, Self Assessment Manikin, emofaces and premo. The Self Assessment Manikin provides three pictographic emotional scales for valance (+/-), arousal and dominance.

• Presence Questionnaire – Witmer and Singer, Measuring Presence in Virtual Environments: A Presence Questionnaire, Presence, Vol. 7, No. 3, June 1998, 225–240

• System Usability Scale – SAS is a standard 10 question questionnaire that rates the usability of a system. Metrics such as the number of errors, time on task, and was the task completed successfully are also commonly used.

• Net Promoter Score – This metric assesses the consumer loyalty for a product or brand. It asks the question “How likely are you to recommend to a friend or colleague?”. The rating is 0-10 with 10 being the best. 0-6 are detractors, and 9-10 are promoters. The NPS is % promoters – %d etractors. www.starred.com is an online tool for use.

• Product Attachment Scale – Four questions that measure a users attachment to a brand or product each on a 7 point likert scale.

  • This product is very dear to me

  • I am very attached to this product

  • I have a bond with this product

  • This product has no special meaning to me

• Cohort trials – Cohort trials look at a single intervention user group in comparison to a user group with no intervention over time. They are best for understanding cause and effect.

• Cross-sectional trial – Cross-sectional study designs are used when studying one or more variables within a given population at one point in time. Such studies are useful for establishing associations rather than causality and for determining prevalence, rather than incidence.

• Behavioral Sampling – Give people a pager or phone and ask them to record and evaluate the situation they are in when it rings. This is a useful way to discover how products and services get integrated into people’s routines in unanticipated ways.

• Be Your Customer – Ask the client to describe, outline or enact their typical customers experience. This is a helpful way to reveal the client’s perceptions of their customer and provide an informative contrast to actual customer experiences.

• Bodystorming – Set up a scenario and act out roles, without or without props, focusing on the intuitive response prompted by the physical environment. This method helps to quickly generate and test many context and behavior-based concepts.

• Empathy Tools – Use tools like clouded glasses and weighted gloves to experience processes as though you yourselves have the abilities of different users. This is an easy way to prompt an empathic understanding for users with disabilities or special conditions.

• Experience Prototype – Quickly prototype a concept using available materials and use it in order to learn from a simulation of the experience using the product. This is useful for revealing unanticipated issues or needs as well as evaluating ideas.

• Informance – Act out an “informative performance” scenario by role-playing insights or behaviors that you have witnessed or researched. This is a good way to communicate an insight and build a shared understanding of a concept and its implications.

• Paper Prototyping – Rapidly sketch, layout and evaluate interaction designed concepts for basic usability. This is a good way to quickly organize, articulate and visualize interaction design concepts.

• Predict Next Year’s Headlines – Invite clients to project their company into the future, identifying how they want to develop and sustain customer relationships. Based on customer-focused research, these predictions can help clients to define which design issues to pursue in product development.

• Quick and Dirty Prototype – Using any materials available, quickly assemble possible forms or interactions for evaluation. this is a good way to communicate a concept to the team and evaluate how to refine the design.

• Role-Playing – Identify the stakeholders involved in the design problem and assign those roles to members of the team. By enacting the activities within a real or imagined context, the team can trigger empathy for actual users and raise other relevant issues.

• Scale Modelling – Use scaled, generic architectural model components to design spaces with the client, team, and or users. This spatial prototyping tool provides a way to raise issues and respond to the underlying needs of different stakeholders.

• Scenarios – Illustrate a character-rich story line describing the context of use for a product or service. This process helps to communicate and test the essence of a design idea within its probable context of use. It is especially useful for the evaluation of service concepts.

• Scenario Testing – Show users a series of cards depicting possible future scenarios and invite them to share their reactions. Useful for compiling a feature set within a possible context of use as well as communicating the value of a concept to clients.

• Try it Yourself/Home trial – Use the product or prototype you are designing in the context it is designed for. Trying the product being designed prompts the team to appreciate the experience the actual users might have.

• Minimum viable product – Rather than build a prototype, use human power and to ‘perform’ the key functions to establish if the core premise is useful.

References

1. http://abacus.bates.edu/~ganderso/biology/resources/stats_flow_chart_v2003.pdf

2. Russell, J. A., Weiss, A., & Mendelsohn, G. A. (1989). Affect grid: A single-item scale of pleasure and arousal. Journal of Personality and Social Psychology, 57, 493–502

3. Mugge, Ruth, Hendrik N. J. Schifferstein, and Jan P. L. Schoormans (2006), A Longitudinal Study on Product Attachment and Its Determinants, in European Advances in Consumer Research, Vol. 7, 641-647

4. https://stoutbooks.com/products/ideo-method-cards-51-ways-to-inspire-design-61457