How to Conduct User Acceptance Testing: Process Stages, Deliverables, and End-User Testing Place in Quality Assurance
While products are built by engineers and designers, they are created for the end user. But, sometimes this idea is neglected. Product design starts by assuming how a product should behave and what it should look like. And while market research and user interviews form the basis for designs, the designs are still just based on assumptions albeit educated ones.
When engaging in various quality assurance activities, development teams are able to look at the product from the user’s standpoint. Such techniques as black box or usability testing help communicate user needs. But, QA activities are aimed at finding bugs and logical mistakes in the software, which are technical aspects of a product. An end user is interested in whether they can actually solve their problem by using the product. They have no desire in getting stuck in a complicated flow. To ensure the development team is building the right product for the actual end users, conducting user acceptance testing is vital.
What is user acceptance testing and how is it different from quality assurance?
User Acceptance Testing (UAT) has various other names, e.g. End-User Testing, Operational, Application, or Beta testing. Those are just a couple names you may encounter, but they describe the same thing. So, UAT is basically a testing activity aimed at checking whether a product being developed is the right one for the end users. In quality assurance, such activities are also called validation, which is a different process in contrast to verification.
Validation (or user acceptance testing) stands for assuring that the product corresponds with business requirements and can be used by the end user. While verification refers to general QA processes aimed at testing the technical aspects of a product, to ensure it actually works.
Validation activity can be divided into two types of testing:
Alpha testing is a type of validation performed by developers using techniques such as black-box testing. This technique assumes testers aren’t able to look at how the system works so they can test it without bias.
Beta testing is the second type of acceptance testing, and its aim is meeting user acceptance criteria. UAT can be performed by the following members in the role of the end user:
- the actual users of an existing product
- users of a previous version of a product
- stakeholders involved in the development of the produce
- business analysts as an end-user specialist
This enables the development team to fix most of the usability problems, bugs, and unexpected issues concerning functionality, system design, business requirements, etc.
Why do you actually need user-acceptance testing?
It’s important to mention that UAT isn’t tailored to reveal technical/design bugs in the existing software, but it doesn’t exclude finding some. The purpose of acceptance testing is to validate that the product corresponds to the needs of the real users and is ready for launch. According to an Origsoft survey on UAT usage, over 75 percent of respondents said they conduct multiple cycles of end-user testing with 57 percent of them stating the poor quality of the product as a reason.
Your development team can be close enough to deliver functional applications, but never see the picture through the user’s eyes. So, before the production stage, it’s much easier to find and fix possible issues, rather than losing thousands after launch fail.
Nevertheless, UAT requires organization and preparation work to make it effective. If you want to ensure your product’s validity, consider the following steps in conducting user acceptance testing.
1. Analyze product requirements and define key deliverables
The first step is looking at your functional and business requirements and figuring out what will be tested and what people you need to get valid information from. Regardless of the methodology you use, analyzing requirement documents will be the first step, as it gives you input information before proceeding to the actual end-user recruitment.
The area of your interest is business and functional requirements. Business requirements are the high-level objectives of your organization that communicate business needs. Those may sound like “customers should be able to use multiple payment methods.” Functional requirements bridge a technical solution with the business requirement. So, the functional requirement would sound like “implement PayPal, Visa & Mastercard, Payoneer payment gateways.”
The overview of these requirements will tell you exactly what you should test, if the implemented solutions work for the users, and if they solve the problem(s) for business. Functional requirements can be translated into test cases, considering the success criteria of business requirements. So that will help you to form an overall testing strategy. Consider engaging your business analysts, QA engineers, or product owners for requirement analysis.
The final planning stage is creating technical documentation for the UAT process. Here you want to document your testing strategy, rules, test scenarios/cases, standards, etc. The following sections describe the documentation used in user acceptance testing.
User acceptance testing deliverables
The planning stage is finished when you have a plan of action. Creating a UAT test plan will help you to keep everybody aligned with the same objectives and vision. The UAT test plan is the main document that includes all the information concerning what will be tested, by whom, and how. It is essential to cover all the organizational and processual aspects of UAT, as long as you work with the users, not QA engineers. Here are templates of UAT test plans that can be used for your project:
- Test Plan template by Coley Consulting
- sfsu template
- iiba template
- iiba template
The following points should be included in the plan and basically describe low-level details in the process:
End-user testing strategy. This is your strategic approach on what to test, how to test, and who should be tested. Here you want to describe the product you are testing, the purpose of user-acceptance testing, types of tests, and objectives. Your testing strategy should cover such information as:
- Product description
- Testing objectives
- Testing scope
- Standards
- Testing types
- Testers/roles
- Process curators (managers)
- Reviewers
- Reporting standards
- Outcomes
Entry criteria. Conditions that define the software is ready to be tested. These conditions are defined in the stage of planning by the development team, QA, business analysts, and stakeholders.
Exit or acceptance criteria. Conditions that define the software is valid for the users. These conditions would be the final stage of your UAT. To understand if a product is production-ready, you should assign reviewers that will analyze testing results and define whether user-acceptance criteria are met.
Test scenarios. Test scenarios are written to validate that the system is usable, checking the end-to-end operations with real data. To cover all the common use-cases, test scenarios can be approved by the product owners, business analysts, or real users. Each user scenario is connected with one or two requirements or user stories. A test scenario should convey a simple idea of what will be tested. A simple example of a scenario is “check shopping cart functionality.”
To write good test scenarios for user acceptance testing, consider involving end users in approval to include all the possible use cases. Also, consider writing them in plain language, avoiding complicated phrasing, or overly techy explanations. To speed the process, you can use a template for test scenarios.
Test cases. Test cases should correspond with all the test scenarios. Most often you will convert your user stories and business use cases to write efficient test cases. Examples of test cases are:
- Check unregistered user adding the product into the shopping cart
- Check shopping cart filtering
- Check the “continue shopping” button
Test cases are efficient when there is a clear purpose stated, and the user is able to understand what they should do to complete it. Making it easier for the end user to read and perform a test case may look like this:
- Open the application
- Add any product to a shopping cart
- Authentication is not needed
- Proceed to the shopping cart
You may also include expected results in the test case, so the user is aware of what is going to happen:
- The product will appear in a shopping cart
- The system will ask you to authorize as a registered user
Test cases may be written differently, but you can use a free template.
Reporting standards. Description of what a report should look like and what information an end user should provide.
Test reports. These accumulate documented output data when the test is completed. Depending on the testing standards and testing scenario, reports can be satisfied with various information. But, most often, in UAT, QA teams will require only a sign-off from the tester. A sign-off is just a confirmation that the test is successful and it corresponds to the user’s criteria. See a test report template for an example.
At the end of UAT, deliverables provided can be used by QA engineers or a UAT manager to extract valuable data and communicate results with the development team.
Traditionally, quality assurance engineers will be responsible for end user feedback processing. The results of tests, bug reports, and fail/pass records are provided to the developers to ensure constant communication between different parts of the team. Based on the end-user feedback, the QA team can also provide software quality metrics to measure progress in terms of UAT.
2. Choose the time and form of end-user testing
Acceptance testing may take place at the very beginning of the project, or it may be the last step before the actual release. This depends on the methodology you are using for the project. As two of the most popular project management methodologies in software development are Waterfall and Agile, we will look at the user acceptance testing process within those two models.
Acceptance testing in the Waterfall model
To dive deeper into the details, we need to quickly recap what a Waterfall model is. A Waterfall model is a traditional project management methodology based on a step-by-step development of the product.
The stages don’t intersect, meaning there is no simultaneous designing and design testing, or development and testing. The whole process is strictly documented and is aimed at delivering a fully functional application at the end of development without iterations.
User-acceptance testing in Waterfall takes place at the final stage of development, right before the launch. It can be conducted only after the system passes all the other preparations and is considered ready code- and function-wise. Acceptance testing can be started only if the following check-points are passed:
- Complete business requirements of the product
- Finished code base
- Finished QA activities (System, integration, unit testing)
- Bugs revealed during the QA stage are fixed
- Minor visual issues are acceptable
- User-acceptance environment (UAT manager, tools for testing, test scenarios, etc.)
In the Waterfall model, user-acceptance testing is a definitive point that shows software readiness. If a product passes user acceptance criteria, it means the product is ready for production. UAT activities, in that case, are aimed at the full check of the system, its functionality, usability, and bugs. But the primary aim is to ensure that the product corresponds to the initial requirements and end-user needs.
User acceptance in Agile methodologies
The Agile model of software development isn’t as straightforward as Waterfall. It’s based on iterating each development stage until the product reaches the required quality and functionality. Iterations of each phase allow for highly flexible development and dynamic change in requirements, as Agile doesn’t focus on creating much documentation. That allows the development team to quickly respond to changing requirements from the customer. But, to ensure the validity of the product for the end user, UAT can also be conducted within the Agile model.
The graph shows the Agile product development cycle with iterations at each stage. User acceptance testing can be conducted at each stage of the project. The main difference between UAT within the Waterfall model and Agile is that end users may impact the initial requirements in the course of iterations. As acceptance testing is present across the entire product development, it also impacts the overall quality of development, as it delivers instant feedback on what works the best.
Required checkpoints to start end-user testing in the Agile project is:
- Formed business requirements
- UX/system documentation
- Testing material (interactive mock-ups, high-fidelity prototypes, demos)
- User-acceptance environment
Further testing is held in the course of each sprint/phase. Depending on the specific phase, UAT may take different forms and use different tools. For example, these can be tests on functional and non-functional requirements, or early-stage testing to validate assumptions made during the planning. At the end of each iteration, acceptance testing produces deliverables that are used to modify requirements/system architecture/UX style guides, etc.
3. Recruit users and form UAT team
As we mentioned earlier, testers can be recruited from your existing user base. Depending on the project specifics, those can be subject matter experts, leisure users, stakeholders, business analysts, or the customer. To recruit the right people, contact your sales department to find the actual users. You can also use crowd-sourcing platforms to search for the testers or hire a freelance user-testing specialist.
A good practice is to create a social media message, or even a landing page to draw the attention of your audience. The testers should not be tech-savvy or acquainted with the testing processes. If they have used/will use your product, or had experience with a similar one on the market, they will suit your needs. That will help you to eliminate the need for deep onboarding and QA team involvement.
4. Implement end-user testing tools and onboard testers
Of course, there are specific instruments on the market that are designed for end-user testing. We might suggest professional QA systems to your testers, but they will cause a lot of confusion for non-IT workers. The tools listed below offer testing management functionality like reporting, task overviews, and testing documentation templates:
Usersnap is a popular platform for providing visual feedback on the tested software/web-based applications. Basically, it’s a tool that allows users to mark the bugs right on the screen, leave comments and suggestions, and share the feedback.
Testgoat allows you to start with the testing right away, as there are templates for UAT deliverables that can be downloaded and modified. Testgoat also offers reporting capabilities, to create bug reports from test steps, or from scratch. It also allows you to assign reported bugs to specific people or teams, which makes the process a lot more organized.
Bugwolf is another instrument for conducting UAT. Besides testing environment and bug reporting, it offers gamification and competition testing capabilities. You will also find useful built-in payment options if you are going to conduct end-user testing online.
Well-known project management tools like Jira or Trello also have functionality for conducting UAT.
5. Create user acceptance environment and conduct training
To get the most out of end-user testing, start with the training. Your testers and UAT manager are responsible for that. Consider structuring your training process to include the following things:
- Introduce users into the testing process and its objectives
- Train users to use tools for end-user testing if you are going to use them
- Provide them with reporting standards and guidelines
- Ensure users understand test cases properly, providing support if needed
- Supply them with access to the testing environment
Most often end-user testing can be done on the user’s side, meaning you won’t have to supply your testers with the hardware. The whole process can be done online. More complicated projects or confidential data may require gathering a dedicated team of user testers at your office. It’s also important to appoint a manager and give tasks to your QA engineers to provide documentation, tools, and support.
6. Run the tests
Once you have your test scenarios and test cases, you are good to go with the tests. To support your end users during the process and get the required results, deliver a clear understanding of what actions each test case requires. Keep in mind that your users are not professional testers. During the test, be sure to provide real or close to real data to the users, avoiding sample content or dummy buttons. Any misconception may get them stuck at the test case.
Another important aspect is having your developers at the ready to fix anything that goes wrong. Your testing environment can shut down, or there can be errors preventing users from testing. Users should be able to access required functionality at each stage of testing, whether it’s an interactive design or a functional app, to allow them to perform each test case included in the test plan.
7. Collect output information and analyze it
At the final stage of your UAT, you will get tons of data from the testers. Your QA team will have to analyze it. The data is collected via user reports submitted manually or via a specific tool. Additionally, you can conduct interviews with separate users to get more insight about the test cases they performed and what they think of them.
To evaluate the system readiness, you may consider measuring the percentage of tests passed/failed/fixed. There are also a few more points that should be considered:
System stability. The stability can be determined by the number of unexpected errors met during the UAT.
Coverage of testing. The coverage is measured by the number of test scenarios/cases written and their ratio to the overall finished tests. You may also match your UAT testing results with the user journey map to understand which part of functionality was left untested.
Usability of the system. This can be calculated by the number of tests not passed because the user didn’t find a way to do it. But the overall UX is tested during the usability testing, which is conducted as a separate activity.
Contract/requirement compliance. Requirement compliance is checked after all the end-user tests are finished. It ensures that the software build still corresponds to the initial requirements/contract scope, even after changes brought by user acceptance.
When the acceptance criteria are reached and approved by reviewers, the product is production-ready.
To sum up
Testing your product with real users before launching is a good practice to reduce the number of overall defects. While end-user testing won’t give actual solutions to the problem, it will help to reveal the problems your developers or QAs couldn’t think of. More importantly, the defects found before the production stage save your budget, as the product goes to market in a close-to-perfect form.
Additionally, depending on where you put end-user testing, development costs may be saved. Conducting UAT at the stage of system design will help to reach the agreement with the stakeholders by allowing users to decide what works for them before the actual software is built.
Originally published at AltexSoft tech blog “How to Conduct User Acceptance Testing: Process Stages, Deliverables, and End-User Testing Place in Quality Assurance”
