How we created a measure to demonstrate the impact of design
When the design and user research teams initiated the discussion about measuring design maturity at Zalando, there were more questions than answers. It was apparent that product design and user research had an impact not only on our product, but also our work environment. The teams could look at customer satisfaction to gauge our impact on the product, but how could we measure designers’ impact on the overall work environment? And how could we identify which factors influenced the extent of our impact?
We* tried several of the available maturity models and scored well for overall design maturity, but we wanted to know more — how could we measure maturity on a unit or team basis? And how could we measure the impact product design and research has on the way other Zalandos think and act?
…how could we measure designers’ impact on the overall work environment? And how could we identify which factors influenced the extent of our impact?
With these questions in mind we embarked on a journey to discover the best way to measure design maturity at Zalando, how to derive the value of design as a capability and finally, how to measure progress over time in the various parts of the business.
The measuring and benchmarking of design maturity has been an ongoing topic within the larger design community. One of the most well-known models is Nielsen’s model of corporate UX maturity (see here and here). This and other “early” models primarily seek to measure maturity on an organizational level, using several stages of maturity based on the capabilities an organization demonstrates (e.g., whether a dedicated usability team exists). For an overview of existing models see here.
While these earlier models are useful for giving us a birds-eye view of the development within an entire organization (no maturity vs. very mature), they are not granular enough when it comes to identifying maturity and non-maturity levels among individual teams. This latter measurement is particularly interesting for us here at Zalando given that design and research capabilities are not evenly distributed across the organization. Therefore, we would expect different maturity levels across the company which cannot be demonstrated with an approach that only shows the organization as a whole.
Newer design maturity models or measurements of customer centricity tend to look at the level of innovation in companies, and focus more on the individual level (e.g. Artefact’s Design Maturity Survey or IDEO’s Creative Difference Survey). Using surveys that ask individuals about their attitudes and behaviors, and matching this data with information about organizational structures, gives a fuller picture of individual maturity levels and the settings in which they evolve. Sony’s Design Driven Culture Approach is an even newer concept that measures maturity (although in the context of company culture) by focusing on the thinking and doing of individuals as a indicator of overall design maturity. This model also tends to focus on the organizational and spatial structures that enhance maturity.
Based on our desk research we were able to to compose a set of assumptions that guided our search for a design maturity model at Zalando:
- Overall design maturity is thought to lead to better and more human-centered products thereby increasing customer satisfaction and revenues.**
- The values and purpose with which product designers identify will manifest in their daily work and communications, thereby influencing the non-designers working around them.
- Proximity is key — the closer and longer one works with a designer, the greater the impact.
- The more a non-designer can think and act like a designer, the more the non-designer will be able to use designers’ skill to address strategic questions — and earlier involvement of design thinking will lead to more customer-centric products.
Compiling these assumptions made it clear that our model should combine measurements of organizational set-ups in which design thrives (which we know from existing maturity models), with measurements of individual maturity. With regards to individual maturity we made the decision not to simply measure the maturity of individual product designers, but rather all members of the organization, regardless of job title. For this, we needed a standardized survey.
Overall design maturity is thought to lead to better and more human-centered products thereby increasing customer satisfaction and revenues.
With the use of a survey, we could learn how our design and research work influences the thinking and doing of our fellow Zalandos over time. However, we weren’t big fans of the buzzwords “thinking and doing”, and therefore decided to rename thinking to design mindset and doing to design excellence. Our next challenge then, was to define the measurement for design mindset and design excellence, as well as the organizational set-ups that should have an impact, as defined in our assumptions.
Because the survey’s content would have to reflect the work and values of the product design team, we knew we would first have to identify the purpose and capabilities that unite the designers and see how these could be translated into the thoughts and behaviors of non-designers. Luckily for us, these capabilities are clearly defined as part of the design job family at Zalando, and we were able to use them as our starting point. Based on this we could identify empathy, craft, experimentation, and sharing as central components of our measurement. These terms are also part of our design purpose statement: We combine empathy, science and creativity to make engaging and joyful user experiences. This added an even stronger point of using them as our basis for defining the dimensions we wanted to measure.
Our next step was to operationalize these capabilities and bring them to life in a measurement. For that, we thought about how each quality manifests itself in the designers themselves, and how a high maturity level in designers could manifest in non-designers’ mindset and excellence by working alongside them.
When measuring empathy, for example, we created survey questions concerning the following:
a) 360 degree view of users by leveraging different kinds of data.
b) challenging biases in your work (especially those about your users).
c) having the curiosity to really understand a problem.
We’ve found these behaviors to be standard for designers and researchers, and also consider them desirable and achievable for non-designers and non-researchers. One way of assessing these qualities in non-designers, for example, is to ask them when they last interacted with users.
…by measuring maturity on a individual level, while also tracking the organizational structure individuals are tied to, we are able to identify opportunities for improvement.
After this exercise within each of our dimensions, we developed an initial survey with A LOT of questions. We ran it as a pilot in paper and pencil form — directly interacting with participants in order to gather any feedback and reactions. We analyzed the quantitative survey results in an exploratory factor analysis in order to understand which of our questions were best at measuring the constructs we had defined. Based on this understanding, we selected three questions to gauge design mindset, and three for design excellence. These six questions have become our standardized measurement, which we continue to administer online every six months and measure against our baseline survey. We also ask additional questions regarding the team’s setup, perceptions of design, and opinions about our design system in order to discover ways to maximize the impact of design throughout the entire company. Therefore, by measuring maturity on a individual level, while also tracking the organizational structure individuals are tied to, we are able to identify opportunities for improvement. In addition, this gives us an opportunity to understand the reasons behind low maturity (why, for example, one team is employing design mindset and excellence to a higher extent than another).
Opening survey data is often like trying a praline — you never know what you’re going to get (hooray for the Forrest Gump analogy here!) It was the same for us — we were surprised by the open and honest feedback, and the praise and encouragement, but also the numbers. In mediation analyses, we uncovered how the number of designers and amount of research are related to maturity — giving support to our assumption that proximity to design and research play a central role in building maturity. We learned how various parts of the organization leverage design in different ways. And finally, we learned that management is even more excited about design maturity numbers than we had anticipated.
One of the first tangible outcomes of our work was the creation of a training program for user research, having found that this is one of the most straightforward skills to nurture in order to increase design maturity. It’s also great that we now have the data to show the evolution in the understanding and usage of our design system.
For now the journey continues…we are working on increasing our response rates to the survey so that more people will participate and share their thoughts. We’re working on another iteration of the survey in order to improve our measurements. And, we’re spreading the word, showing what individuals or teams can do to improve and celebrate design maturity in teams.
*the “we” includes Rob Manzano (team lead of user research for Zalando)and Anne Pascual (vice president of design and fundamentals for Zalando), in addition to the author.
**this study shows a correlation between these constructs
Dr. Franziska Roth is a senior user researcher with a quantitative focus for Zalando.
Thanks to Eileen Bernardi for her support on writing this story and to product design leads: Jay Kaufmann, Lorenzo Fernandez, Gloria Rupprecht, and Ray Ho who all lended support in the process of measure development.