“These commissioning constraints not only have a direct bearing on the decision as to which methodologies to use but also how the evaluation design is created.”

UX/HCI Constraints

I’ve recently been thinking more about the real world problems of UX and HCI evaluations for a book chapter I’m writing on how it’s really done.  Textbook approaches to experimental design, and the choice of research methodologies, often rely solely on selecting the best methodology for the job at hand. That methodology, assumes a certain knowledge of the application domain, with specifically favoured methodologies becoming popular in different domains. In reality, and certainly within the human factors domain, things are not so straightforward. The real-world inflicts a number of constraints at this point and so unless the experimental design is for academic work, which you have full control over, aspects of the design will be beyond our control. I have found that we are often approached by third-party organisations which do not have the usability, and human testing experience that we possess. In this case, there is often no design requirement to use a certain research methodology, but there are limitations of time and budget allocated for the specific work.

In this case I’ve come up with a set of commissioning constrains which seem to represent the spectrum I’ve experienced:

• Just-In-Time Constraint. Often commissioners tend to leave aspects of the project planning which they know least about until the end, preferring not to go too deeply into this area because they become worried about their own lack of understanding. Therefore there is a tendency for the UX specialist to be approached just in time for the user testing to begin. This puts incredible pressure on the  specialist to design and instigate testing on a very sort notice;
• Too-Little-Time Constraint. Extending the Just-In-Time Constraint, the Too-Little-Time Constraint suggests that in most cases, ‘user evaluation’ is all that is included within the project plan and the finer points of understanding how this testing should proceed are neither understood or considered further. In this case there has not been enough time allocated for post tests or non-linear aspects of the work which could be run in parallel with other aspects of the project plan. As enough time has not been devoted to these aspects within the plan, testing is likely to fail or at best be very rushed;
• Non-Specialist Constraint. There are additional variations of the two constraints previously listed; being those which include a very badly planned user evaluation, usually by a non-specialist, whereby because the planning stage is inflexible to higher-level management a non-scientific experimental evaluation looks like it needs to be considered;
• Inadequately Funded Constraint. This constraint comes to the fore when user evaluation seems to be planned reasonably well but the amount of funding required to accomplish it – and participant payment within that funding model – are not adequately considered;
• Pre–Supposed Outcome Constraint. Planning in the real world often focuses on the achievement of a desired outcome. Indeed, most project planning and business planning revolve around achieving the correct outcome at each stage of the process. Therefore a manager or engineer planning a piece of work will have a desired outcome in mind, which in reality will not just be a desire, but be a project requirement. In this case there is an enormous amount of pressure on the UX specialist to deliver this requirement, support by the testing or not;
• Implicit Overrun Constraint. Even when there has been enough funding and time allocated to an unplanned user evaluation, it is often accepted that it will be the final stage of the development and therefore the plan is seen to fit into a linear dimension as one of the last parts of the project development. In this case, overruns in other aspects of the work, often considered to be more important than the usability testing, tend to eat into the time available for the linear plan;
• Due Diligence Constraint. Normally as a function of the Inadequately Funded Constraint and in combination with the Too-Little-Time Constraint we see that in some cases the usability study is only an afterthought such that the software engineers can demonstrate due diligence; and in reality expect no more than a cursory ratification of the development. Here we can see that this is very difficult for the HCI specialist a handle because acceptance often implies a Pre-Supposed Outcome. How this is handled, and whether the contract is undertaken by the specialist is a matter of personal choice; and
• UnConstrained Experiment. Finally, there is that rare occasion whereby the work of the specialist is understood and respected by the project planner, and indeed be commissioner of the development. This is often the most rewarding type of work undertaken and in many cases occurs in academia or in research and development work for large corporations whereby ergonomics, human factors, and human behaviour are the central focus of the outcomes which are expected as opposed to a standard software development. We describe this, in the our Laboratory, as the unconstrained experiment. Here we can bring a well-designed experiment using multiple methodologies and hybrid techniques to bear upon a problem domain using a full range of tools instruments techniques and hypotheses.

These commissioning constraints not only have a direct bearing on the decision as to which research methodologies to use but also how the experiment design is created.