Steve Roberts

Balancing opinions and data in User Experience design


Big Clients, Design, Strategy


User Experience Design, User Research, User Testing

User Experience (UX) is where I spend most of my time in web and mobile development. Designing/Creating/Architecting/Planning the product all involves working on how the user will experience the product.

The most difficult part of this work is that everyone in the business who has even the remotest connection to the product development project has a very strong opinion about it, because they can imagine being a user ergo they know what users want. This results in a lot of ideas for the product, which in turn means scope creep and cost and time blowouts or feature bloat - a product that tries to do too much.

Qualitative UX

The opinions and ideas, as well as any market research or other product related 'evidence' that stakeholders provide are valuable to the product design process. The job of sifting through this material and designing a beautifully suitable solution is what a UX person does. This is what I would call the Qualitative or subjective part of UX design/planning/architecture. Each piece of information is evaluated in context of the product idea and through discussion (lots of discussion) the UX person forms their opinion of what form (if any) each piece of information should play in the proposed solution.

Selling the proposed solution as the best way forward is difficult, as it is just one way of many possible and worthwhile ways forward. Rhetoric, metaphor, analogies all come into play in the process of persuading the decision makers that the proposed solution is the one we should go with, but there is always a risk that this won't work.

If a proposed solution doesn't grab a group of decision makers then trouble can occur. The decision makers themselves may want to jump in and offer alternative solutions. This can become a dreadful 'design by committee' situation, where the solution that results is merely a collection of the most popular ideas as voted by a group... eugh!

What is needed at times like these is data, quantitative data.

Quantitative UX

We live in a fast paced world that has less time for nuance and prefers more easily understood black and white choices. This is terrible as the process of reducing everything to easily understood good/bad inevitably means that something is lost . This applies just as much with web and mobile product development. Reducing a product down to more understandable items that can then be measured and categorised into good/bad (or in scope/out of scope) can mean that the bigger picture of what the product is or should do can get lost (see 'design by committee' above).

At the same time, reducing products into elements and rating each element - essentially turning the product into numbers can be the only way to get a solution out because in the fray of discussion that product planning meetings can become, numbers can cut through opinion and show a clear way forward.

It's about the quality of the numbers, stupid!

There are bad numbers, and there are good numbers. Anyone can come up with a model that produces an impressive graph at the end, and I'm sure there or many examples of such graphs winning over decision makers. However I prefer to be prepared for someone to question my methodology when I present an opinion backed up by data, so there are only certain numbers I will stand behind. Here are some categories that I would avoid as they can be too easily questioned (contestable) and some that areĀ  valid in the context of UX design (incontestable).

Contestable numbers:

  • Market research data
  • Data from a product in the same stable

Incontestable numbers:

  • Usability testing - tasks successfully completed or not?
  • Hueristic analysis - judging a design against to a set of UI categories
  • Product analytics - product objectives meeting targets or not?

I'm avoiding the use of 'good' and 'bad' here because obviously market data and data from a similar product are very useful. However these are contestable because there are too many variables in play for you to say something like, 'In our market research 86% of people said they $5 is a bargain for product X, therefore we should make price a strong element on the product purchase page'. The variables here are things like; what else is on the page that may be more/as important, where these visitors came from, if they are browsing and therefore want research first... more than enough questions that although they could be addressed can easily water down the effectiveness of using the numbers in the first place. Likewise analogising from a similar (successful) product is fraught with danger, for example 'It will work just like the wall in Facebook'. Your product is not Facebook, you don't have users who are 'interface educated' as much as Facebook users or who will necessarily use your product at the same time, or for the same reasons as they use Facebook (i.e. when they are at a loose end for 5 minuttes at work) and it is very unlikely that your product will have the depth and breadth of user generated content that appears on most users Facebook walls.

So these types of contestable data are things that I would use as part of my rationale for presenting a user experience design that is my professional opinion. I wouldn't push this opinion as an absolute imperative until I had incontestable data that backs it up.

Of the three incontestable data types above, unfortunately only one is available to a UX designer at the point where they need it - when presenting their proposed design to a stakeholder group. Product analytics are the best data but they are only available post launch (unless you are redesigning in which case existing analytics can be useful but watch out for mistakenly analogising apples with oranges). Heuristics, such as these are great but they are bit like judging whether a site is accessible or not, a lot of the WACG debatable as to whether or not your site passes/fails. This subjectivity is lessened though because of the aggregation across multiple heuristics - your stakeholder may argue with you on 3 of the 10 points but that still leaves you with enough data to add weight to your opinion of whether the proposed design is a good one.

By far my favourite method of backing up my designs is usability testing. This data only becomes available once you have enough of a design to test, which is sometimes not available but even testing against wireframes has some validity. Usability testing can be done very simply by defining a set of tasks that your users will need to complete in order for the product to achieve it's objectives. For example the tasks for users of my site should lead to the objective of generating leads for consulting work.

  • Search for someone to help you with the web strategy of your small business
  • Find an article on the site that interests you
  • Contact the site owner

Obviously my site needs to have a lot more content on it for these tasks to be completed by most visitors to my site, but by getting a tester to sit down and perform these tasks - by searching google and then by pointing to parts of a print out of a wireframe of my site's intended layout - I will quickly gather enough data to be able to say that the design will or will not succeed in enabling users to complete those tasks.

This puts any stakeholder discussion of your design back where it should be, clarifiying the objectives of the site and highlighting whether there are content gaps. This is a much better place to be than having a room full of people with ideas of how to design a product - trust me.

Posted in Big Clients, Design, Strategy. Tagged with User Experience Design, User Research, User Testing.

Published 27 May, 2011