Page 7 of 14
Longitudinal evaluation
For products and services that are used over many days, months or even years, it becomes important to understand the use of the product or service over that length of time, not just one-off use.
This might be relevant in the case of:
- The product fulfilling a long term need – e.g. a pacemaker
- The product use changing over time – e.g. emergency support when a chronic condition worsens
- The product changing over time – e.g. app based technology that changes based on need
For products and services that are used over time:
- User needs that the product meets may be expressed in relation to time
- Risks associated with their use may be expressed in relation to time
- Longer term engagement and use may change over time
In any of these situations it therefore becomes important to understand what options you have to understand how your target users will use your product over time.
Benefits of longitudinal research
Many of the stages of testing outlined below can take place at a single moment in time and can also take place in a lab. However, if possible, we at Ctrl Group would typically favour longitudinal research in context. While we’ve outlined the benefits of contextual research in Module 2, the additional benefits of longitudinal research are:
- Identify misuse or malfunction over time
- Identify changes in use over time
- Identify habits formed over time
- Understand how product changes in line with use
- Understand behaviour or personalised features over time
Challenges of longitudinal research
Despite the benefits of carrying out longitudinal research it is also important to understand the challenges from the outset. The advice of Ctrl Group is to make explicit both the benefits and challenges of this type of research for your project, and use this to inform if and how you evaluate your product longitudinally. In short, the main challenges we find in carrying out longitudinal research are:
- High dependency on using a high fidelity prototype or functional version of your product that can be operated autonomously over a time
- Getting the timing right so that the product and research are ready to start at the same time
- Recruiting and consenting participants willing to both use a test version of a product and be monitored longitudinally
- High resource costs in managing and monitoring a product in life
- Creating a research environment that is a realistic and useful longitudinal assessment
- Managing increased risks associated with autonomous or live product variations
Types of longitudinal research
While a number of the research approaches we discussed in Module 2 (such as ethnographic analysis and diary studies) are longitudinal, when we look to validate whether a product or service meets the user needs we typically work with a higher fidelity product or service. Often the service tested falls into one of the following categories - all of which can be referred to as ‘user acceptance testing’:
| Stage of testing | Purpose | Notes |
|---|---|---|
| Early functioning testing: | Test specific functions over time with a friendly group. | This can be carried out with internal team members and can happen in any context. |
| Alpha testing: | Test of an early version of full or minimum viable functionality. | Can also be carried out with non-representative audience but is typically carried out with users who have not been involved in developing the product. |
| Beta testing: | Test a launch product candidate with target users. | Typically carried out with representative users in the natural context of use. User expectation is often managed with reference to the product as a beta. |
| In-life testing: | Assess a product in market following a live launch. | This assessment happens following product launch and can involve the entire user base or a subset of the base that has agreed to be part of ongoing research. |
| A/B testing: | Assess variations of a product live in-market by managing which elements of your product users are exposed to. | Only possible to carry out if you can manage live variations of single product and gather feedback from target users. This approach is used to assess personalised features. |
Resources
Managing complaints
When you’ve launched a service for real world use, whether or not this is part of the research process, you also need to consider how to manage complaints that may arise from participants. These can also relate to adverse events
When they’re the first point of contact, the research team (managed by the research lead) should record and assess any complaints against their research practice. They need to decide if the complaint is related to how the participant has been treated while taking part in the study or whether it relates to a serious event in relation to the study procedure (e.g. a serious adverse event).
You should agree an approach with the participant that outlines:
- How the complaint will be dealt with
- An approximate timeline
- Who’ll review the complaint
- Any immediate action that can be taken to correct the situation
Document this approach in your records.
The complaint should be reviewed and the findings approved by the research lead. He or she, or a designated person, should contact the participant (in person, via phone or via email) to discuss any findings and corrective actions that may result from the investigation.
The participant can then decide if they’re satisfied with the way their complaint has been addressed so that no further action needs to be taken, or whether further investigation is required.
Case study
Refining the usability of a health support system through iterative and longitudinal assessment
Challenge
To build understanding of usability issues over time, starting with simple prototypes and developing higher fidelity prototypes in response to user needs and issues in use. One of the specific challenges was for researchers and designers to work together to iterate design stimulus and prototypes between research sessions.
Approach
- We planned iterative research from very early design stimulus to high fidelity functioning prototypes.
- As part of initial contextual research, we asked researchers to capture information and feedback on terminology and feature usability.
- Insights around usability were fed back to the design team as soon as possible. These were discussed, and action plans were made as to what design changes would and could be made. There were numerous trade-offs based on the time needed to make the updates and the research questions we wanted to explore in the next round of research.
- Prototypes that could be left with research participants over time were developed, allowing us to understand what patients were able to do.
- Remote tracking allowed us to understand participant behaviour. This information was analysed before the research sessions and used to prompt and probe the patient understanding of the experience and any usability issues encountered.
- When we’d ironed out most usability and usage issues we created a candidate launch device and used a third party research organisation to user test the device. This testing took place in a usability lab and results were based on task completion and length of time spent on task, for a range of key use cases.
- The cumulative results of all of the studies were used to build a design history that formed the basis of a human factors report.
Results
- Key usability issues were highlighted and addressed early in the process, before large investments had been made to build a working service.
- Multiple iterations of the design were created based on insights and feedback from end users.
- Usability issues based on sustained use were uncovered through longitudinal research and remote tracking.
- Objective assessment of final service candidates was possible by having a third party research organisation carry out final usability tests based on task completion.