Extra! Extra!

Here at Extractable, we are strong advocates of the data-driven design approach to creating powerful, enjoyable and successful web and digital experiences for users. You’ve heard us talking about our experiences with data-driven design on this blog, at conferences and in person, but we wanted to go further and learn about how firms across America, both b2b and b2c, used data in their design processes, what tools they used and what outcomes they generated.

So we commissioned a study from Forrester Consulting to do just that, and the results are now in!

The full study: “Data-Driven Design”, a commissioned study conducted by Forrester consulting on behalf of Extractable, April 2012 is now available for download on our site.

We have also created a great infographic of the key findings, available here.

In this blog post, we will focus on a few key findings and what we at Extractable have learned from the study. Over the next few days, we will dissect the study further with a series of deeper posts on key findings.

We saw two key findings in the study.

Firstly, some 60% of firms surveyed had seen improvements in their website due to use of data. And, if the company also reported they had a repeatable design process, the numbers reporting improvement grew to 71%. This is a powerful result. To us as advocates of incorporating data in the design process, the result is a key validation that the process can produce measurable business outcomes. On a daily basis the projects we undertake for our clients using data-driven design are seeing positive results which we are now seeing across a larger and diverse sample size.

However, the second key finding is that many firms are struggling with data: measuring the wrong kinds of data; missing out on key inputs; and even ignoring key data points.

Some specific examples from the study include:

• Companies don’t know how or can’t apply the right tools and processes to optimize their sites. Only 28% of companies are happy with the tools and techniques they use to measure their websites today. As high as 52% believe there are other tools that could provide them with better insight.
• Many firms are measuring the wrong kinds of data. Sites are often measured on metrics that don’t show business value. For example, 46% of respondents indicated they used “time on site” as a key measure. This doesn’t always indicate a positive experience—it could also mean that users are lost trying to navigate the site.
• Some key data is being ignored. 37% report ignoring data that is uncovered and 34% that they gather data but do not use it. Sometimes this seems to be a factor of the ‘Highest Paid Person’s Opinion’ overriding the insights drawn from key data.

Based on the study and our own experiences, it seems that many firms understand the value of data, are looking for the best ways to use it, but fall short in terms of creating a strong data-driven design process, supported by the right skills, teams and tools to be effective.

After reviewing the study, the strategy team here at Extractable synthesized our thoughts into some core recommendations. To make the most of data-driven design, companies should look to:

• Define the metrics for the site based primarily on the central business goals/outcome of the site (or other digital asset), such as sales, leads, customer service efficiency, etc.
• Measure the right set of data that will allow you to see the effect of the changing elements of the experience on that goal/outcome.
• Include a wider set of data/tools, including behavioral tools as well as data from non-web systems such as financial/sales and customer service tools, to ensure you understand why users are doing what they are doing.
• Apply these insights to the design process. Test the updated designs, analyze the results and repeat.

As a reminder, the full study can be downloaded here:

Additionally, Extractable is hosting a Webinar on the study’s findings featuring guest speaker, Forrester Research Inc. Analyst, Adele Sage. Sign up here.

Here at Extractable we employ a data-driven user experience design approach. That is a rather formal way of saying that we gather as many types of user behavior inputs into the design process as possible, from deep web analytics through user concept testing and full user testing. And once live, aggressively optimize our sites based on actual usage patterns.

In the last few weeks we have been working on a series of contextual user interviews, user concept testing and formal usability testing for a range of clients from including a b2b semiconductor manufacturer, a consumer facing financial services startup and a credit union.

This is where both my frustration and love of users kicks in.  Just on Friday we were sitting with one user (contextual, at his desk) for a couple of hours. At one point he was searching on our client’s current site for a specific document – a document that was highlighted (in bold with a big icon) on top right of the page. He diligently scrolled through pages of document listings below the fold, clicked on every tab and scanning the page as if he were a leopard stalking its prey. All to no avail. He never saw the document icon/link that probably took up 15% of the page real-estate.

Throughout this process I had to physically hold my arm from pointing at the screen and my tongue from shouting out some expletive along the line of “What is that big red thing in the corner”!

Of course I exaggerate my reaction, but it was a frustrating experience as the user was not doing what I, or the rest of our design team, were expecting them to do. And this is why I love data based on real users and their real behaviors. They challenge us to think beyond our expectations of what they should be doing and especially to challenge the perceived wisdom that can exist in many companies,  often of the form, “no, our customers don’t want to do that”. This challenge, knowing that each of our ideas will need to successfully run the gauntlet of user research and testing, pushes us to think wider, deeper and hopefully, more innovatively.

Plus sometimes user research delivers incredible insights, like the user last week that had pretty much downloaded the entire product catalog of our client and all its competitors, and had then built out his own taxonomy to store all the files!  That one session gave us more insight into how their users see the client’s products than many hours of stakeholder sessions with the client’s product managers did.

But, of course, not every session delivers such value.  One testing participant when asked what else she might want to see on the test site answered “More information”. When asked what type of information, she responded “just more information”.   Which takes me back to my starting point, I love users and all the things we learn from them to help us deliver rich user experiences, but sometimes it can be quite a challenging experience for the user experience design team.