” Methods maketh the discipline, and I’d say that UX has some nice native methods in use with some others pulled in from other more traditional product marketing domains spliced up with advertising metrics. Importantly, the most interesting for me are HEARTPULSE which together represent some very innovative thinking which – with minor mods – can be directly applied from UX back to the wider Human Factors CS domain.”
A few months ago Rui made a post on measuring UX and the research methods employed – citing a paper presented by some Google UX Researchers at CHI 2010 as well as presenting a list describing the kinds of methods he proposes:
- Analytics – average time of stay, return rates, aggregate data representation of multiple people
- A/B testing
- Task goals – registration completion, contact form submission, path to purchase
- Customer support responsiveness
- Customer satisfaction evaluation – quantitative and qualitative
- Social sensing – Facebook likes, retweets, google trends
- Experience monitoring – qualitative, representation of a single session
- Mindshare goals – qualitative measures such as awareness, branding effectiveness
- Net Promoter Score
The list follows a previous post in which Rui says:
However, what can actually be talked about is that indeed UX can be measured, directly or indirectly; individually or collectively. And by having the proper metrics, UxD can be leveraged towards the constant improvement of products and services. And this can, I argue, be replicated and generalised across products and services.
Now these methods and this argument interest me – not least because their creation, application, and the inferences made from the resultant data tie into some recent posts of mine. Especially my belief that Human Factors is not Anthropology, Human Factors is not Micro Sociology or Social Science, and Human factors is not Psychology. Human Factors is a different domain with different methodological needs, different analytical techniques, and different standards of interpretation. As I’ve previously blogged, UX seems to come from the product design realm in which ‘intangibles’ are required to become tangible for testing purposes – so that user feedback can be factored into new design. Now lets look at these methods in a little more detail.
- Analytics (PULSE + HEART) + Social Sensing or Net Promoter Score – now these are unobtrusive observational methods which collected and – better still – combined enable us to understand how people feel about a website or desktop application. By understanding the quantity, types, and return rates or users we can infer favourable experiences once we have some social sensing data. My rational here is that analytics provides us with information which is all inferential – people may return to the site not just because they like it but because they have no choice, because they want to complain, because they found it difficult last time. But if people are also tweeting, Facebook ‘liking’ then you can expect that if this figure is say 20% then over 60% will really like the site but can’t be bothered to ‘Like’ it – this is the same with Net Promoter Score. I’d say these methods come directly from CompSci and WebSci and are not much found in other domains to the same extent or used in similar combinatorial ways. ✔ Excellent, Human Factors Native.
- Experience monitoring → qualitative, representation of a single session – how to capture the user experience in a single session; difficult with any degree of accuracy. This could be thought of as the kind of user evaluation method we are all used to; coming to Human Factors from Psychology but then being significantly changed so that it now would suit neither Sociology or Psychology but does suit Human Factors so well that the combination I now consider native. ✔ Excellent, Human Factors Native.
- Mindshare goals → qualitative measures such as awareness, branding effectiveness – in general how much chatter is there – in the media, around the coffee machine, water cooler – about your application or site – lots either means love or hate – silence means mediocre. This is mainly a marketing metric, applied with few changes into the UX domain – indeed there are some obvious similarities between Mindshare and Social Sensing. — Agnostic, None Native Human Factors Methods Applied Slightly Differently in the Human Factors Setting.
- Customer support responsiveness + Customer satisfaction evaluation → quantitative and qualitative + Loyalty – your general purpose quantitative and qualitative interview or questionnaire in which consumer satisfaction can be elicited on a wide scale with deployed resources. You normally find this kind of thing in Social Science and these techniques haven’t changed much in the move to Human Factors adoption – one interesting development is their combination with social metrics such that peer review is provided by giving star ratings to various resources. — Agnostic, None Native Human Factors Methods Applied Slightly Differently in the Human Factors Setting.
- A/B Testing – multiple samples of a test are distributed, including the control, to see which single variable is most effective in increasing a response rate or other desired outcome. This allows us to see how sales can be increased and may include things like size and placement of adverts, as well as the effectiveness of email drives, and different texts within those emails. This method is mainly appropriated from Psychology but simplified for application in the the marketing and advertising domains. Its application into the UX domain follows from the more business focused areas of the domain. ✗ Nothing Human Factors Native Here.
So what does all this mean, well ‘methods maketh the discipline‘, and I’d say that UX has some nice native methods in use with some others pulled in from other more traditional product marketing domains spliced up with advertising metrics. Importantly, the most interesting for me are HEARTPULSE which together represent some very innovative thinking which – with minor mods – can be directly applied from UX back to the wider Human Factors CS domain.