API Testing Approaches and Tools: Postman, Rest Assured, jMeter, and more
Every application you build today relies on APIs. That means it’s crucial to thoroughly verify APIs before rolling out the product to the client or end-users. Some QAs, however, still ignore this layer of the test pyramid, and proceed right to UI testing — at their peril. In this article, we explain why you should do API testing and how to approach it.
What is API testing?
API testing is a set of quality assurance actions that include sending calls to the API, getting output, and validating the system’s response against the defined input parameters, in particular, the accuracy of data and data’s format, HTTP status codes, and error codes.
Usually, API testing is performed on APIs produced by the in-house development team. We don’t test third-party APIs, but we can test the way our software accepts their requests.
The approach to API testing largely depends on the API type. There are web APIs aka web services, database APIs that connect applications with DB management systems, operating systems APIs, and remote APIs for accessing resources located outside the device requesting them.
In this article, we’ll focus on in-house web APIs and their quality assurance.
API in terms of the three-tier architecture
Client, server, and database are the three independent tiers of software architecture. Usually, a client is a web browser or a mobile application that presents the requested info, while business logic is realized on the server side. Client and server communicate through requests based on different transfer protocols. The system and logic of protocol-dependent requests are the API itself.
In the most popular software architectural style for data transfers — REST — clients make HTTP/HTTPS requests to the server. The server retrieves the requested data from the database, transforms it according to the business logic, and returns it to the client in a response format, e.g. JSON, XML, or other formats. Finally, the UI displays it to the user.
Another type of web API is SOAP API — a legacy web communication protocol that is still in use. SOAP isn’t limited to HTTP/HTTPS protocol but supports many others including TCP, SMTP, and FTP, but it works with XML format only. You can find out more about SOAP and how it differs from REST in our dedicated article.
And now, let’s see how it works from the QA side.
Why API testing is important
A basic concept in automated testing is a test pyramid. It demonstrates how to tackle automated testing for a project: what parts to test first as they form the base of the pyramid, and what to leave for the final stages — the pyramid’s tip.
Following the pattern of the testing pyramid, first is the unit test layer — code testing often performed by developers while writing it. Then comes the server layer where API testing belongs. Later, when the front end is finished, UI testing takes place.
As we already mentioned, API testing is often overlooked. But why is it so important?
UI tests can’t cover every aspect of back-end testing and turn out to be insufficient for verifying functional paths and API services. This could leave you with bugs rooted in the server or even at the unit level. But now, the cost of mistakes is way higher as it may require rewriting a considerable amount of code and derail the release.
The solution here is to start testing at the early stages. If the request doesn’t return the needed value at the server layer, it won’t be displayed at the UI layer either. Not relying on UI to be ready, API tests can be performed early in the development cycle. This way, they’ll kill at least half of the bugs before they get serious.
In the Agile processes, where instant feedback is needed, unit and API testing are preferred over GUI testing as they are quicker to create and easier to maintain. While the latter requires considerable rework to keep pace with frequent change.
But don’t think we dismiss UI testing. Certain defects can only be caught by GUI tests and it’s the only test type that truly tests the user experience. So, we encourage you to pay attention to API testing as well and apply it to smooth the overall development workflow.
Types of API tests: aspects of focus
QA engineers approach API testing from different angles. They test whether it returns correct outputs in the expected format, whether it delivers responses within an acceptable timeframe, and how well it integrates with presentation layer software. Testers check an API’s reactions to edge cases (e.g. failures, unexpected or extreme inputs) and potential security attacks. So let’s address each of these aspects.
Functional API tests
Functional testing is the assessment of specific functions within the codebase. It makes sure the API actually works within expected parameters, meaning it:
- returns the desired output for a given input;
- handles errors when the results are outside of the expected parameters.
One of the functional testing types is Positive / Negative testing. Negative testing checks how an API responds to every possible kind of wrong input, while positive testing verifies the correct functioning of the API when the input conforms to the norm. If positive test cases fail, it’s a bad sign, as it means the application can’t perform even under ideal conditions.
API performance tests
To ensure the API can handle the expected or higher load, QA engineers validate its functionality and performance by artificially creating or simulating API calls. We’ll outline the types of API performance testing.
Load testing. The point of load testing is to measure where the limit of system performance under high load lies. That’s why we measure response times, throughput, server conditions, etc., while increasing the number of calls.
Soak testing. Load tests that run over a long period of time can reveal system instabilities like API memory leaks. So when you have a weekend ahead, leave automated soak tests running. On Monday, it will show you whether any unwanted behavior has emerged.
Stress testing. The idea is to gradually increase the count of virtual users to find the point at which the API starts throwing errors, slows down, or stops responding.
Spike testing. Contrary to stress testing, here an API undergoes a sudden spike of users. Spike testing checks whether the API is able to stabilize and return to normal functioning after that.
Scalability testing. You want to be sure that your system performance scales according to the changing load. To do so, increase the number of incoming requests and monitor whether it causes a proportional increase in response time.
Peak testing. Similar to soak testing, here you subject your API to the heaviest load while reducing the time of the attack.
API security tests
Security, penetration, and fuzz testing are the components of the security auditing process aimed at testing an API for vulnerabilities from external threats.
Security testing. It validates whether security requirements are met. This includes authentication, permissions, and access controls, namely:
- What type of authentication is required to use the API;
- How well sensitive data is encrypted;
- What authorization checks are set for resource access, etc.
Penetration testing. Taking security testing a step further, in penetration testing, certain API functions, resources, processes, or the entire API is under attack from the outside. This determines whether the threat vector can be reached.
Fuzz testing. The last step in the security audit tests the API at its absolute limits. Forcibly inputting massive amounts of random data, it tests whether the API will stand it or end up with negative behavior like a forced crash or overflow.
Integration and reliability tests of APIs
Being at the center of integrations between internal or third-party services, APIs need to pass the following tests.
Integration testing. Focused on communication between APIs, it ensures they are well-connected and don’t cause bugs in another API’s modules.
Reliability testing. When synced with various devices, an API should be tested for any possible disconnections.
How to go about testing the API
API testing is performed by submitting requests to the software using the application’s programming interface of the application and then checking if it returns the expected data. When automated, API testing can be easily performed on a regular basis. Its main advantage is speed — less lag time between development and QA, less time spent on debugging in production.
QA engineers perform automated API testing using a testing tool that drives the API. We will discuss the most popular solutions in the next section.
Step 1: Create API testing requirements. To be able to plan API tests, first, you need to determine testing boundaries and requirements. You’ll be answering questions like:
- the purpose of the API and its target consumer;
- the workflow of the application;
- the features and functions of the API;
- other APIs this API interacts with;
- aspects of testing;
- the problems we’re testing for;
- priorities in testing;
- the desired output;
- the definition of Pass and Fail.
Step 2: Establish the API test environment. When you determined the functional scope of your API, the next step is setting up an API test environment which is usually the task of DevOps engineers. It requires the configuration of the servers, databases, and every resource the API interacts with, depending on the software requirements.
Step 3: Make a trial API call. Before diving into thorough testing, make an API call just to check that nothing is broken and the API is operational.
Step 4: Define the input parameters. Plan all possible input combinations. You’re going to use them in your test cases, and later — to authenticate the results to determine whether the API performs as expected.
Step 5: Create API test cases. After all the preparations are done, you can write and execute test cases, and, in the end, compare actual results with the expected ones. A good practice is to group them by test category. Examples of API test cases include:
- testing value in response to the input condition: You define the input and authenticate the output, which can be any type of data or status (e.g. Pass or Fail);
- checking the behavior of the API in the system when there is no return value in response;
- tracking events or interruptions triggered by the output;
- authenticating the effect on the system after data structure updates;
- validating resources modified by the API call.
Tools for automated API testing: top choices
API testing isn’t very different from any other testing. But, it requires specific tools, dedicated to automating it. They vary depending on the programming language they’re based on.
Using an API testing tool, QAs usually either take advantage of its out-of-the-box solutions or develop a customized framework from its components.
Postman is currently one of the most popular tools used in API testing. Started as a browser extension for API validation, today with integrated test automation features, Postman is much more than just an HTTP client.
Postman advantages. The tool supports many integrations. It has a friendly UI for constructing requests and reading responses, which allows for creating automated tests quite fast. Postman runs on local machines, so you can stay in control of your data. Using its command-line tool Newman, you can integrate these tests in continuous integration (CI) environments.
However, Igor also warns that it’s hard to maintain large API collections and its built-in versioning system is weak, so, “ It is quite painful to work on the API tests together with the whole team. “ Igor’s recommendation for large projects is to create powerful API tests with additional libraries like Requests (Python) or REST Assured (Java), which we’ll describe in a bit.
Postman pricing plans. Postman has a free version for small projects. If you need access to professional collaboration and extended features, you’ll have to pay $96 per user annually for Postman Pro. There’s also Postman Enterprise with advanced features and extended support, which will run you $216 per user a year.
Requests: human-friendly HTTP requests in Python
Requests, an open-source Python HTTP library, was released under the Apache2 License with the aim of simplifying HTTP requests.
Requests advantages. The Requests library saves time and effort by fully automating keep-alive and HTTP connection pooling. No need to manually add query strings to URLs and form-encode POST data. Igor Pavlenko considers Requests a powerful library with easy-to-understand documentation, simple syntax, and rich functionality.
“It supports such features as OAuth 2 and Open ID Connect Authentication, various Certificates and much more. So, the choice is simple — if you have Python — Requests is the best decision for you.”
REST Assured: a Java library supporting XML and JSON requests
REST Assured is a Java library for creating a REST API testing tool script. So, to get started with it, you’ll have to set up a new Java project first, and then include it as a library for your project. Its documentation is stored inside the Github repository. REST Assured is the best fit for functional testing of REST API services and it requires strong coding skills to create tests.
REST Assured advantages. Rest Assured functionality is limited. However, if you only need to check request and response, the tool will simplify things: In REST Assured, you don’t have to create boilerplate code to test and validate complex REST responses. The tool is open-source — another big pro.
jMeter: a Java-based tool measuring SOAP/REST API performance and more
Apache jMeter is a time-tested tool initially used for load testing. These days, Apache jMeter also supports functional, regression, and stress tests on different protocols. It’s a free open-source tool with custom scripting functionalities that require advanced programming skills.
jMeter advantages. jMeter’s pluggable timers allow for simulating various types of loads. Its plugins add extensibility and customization, while built-in functions enable dynamic inputs to a test. Thanks to its efficient UI design and lightweight component support, jMeter executes tests fast and provides accurate timing. It caches test results and data providing their offline replay and analysis.
Oleksandr Fomenko, a Senior Test Automation Engineer at EPAM Systems, choses jMeter to measure the API performance for its convenient way of multi-threading emulation. “ You can create test cases using jMeter’s record and playback technology, extract and reuse data from response, verify it using JSON Assertion, debug it, and build a comprehensive report. With these functions in stock, JMeter can be easily used as a stand-alone framework for testing REST APIs.”
RestSharp: simple RESTful API testing in C#
Available as a NuGet package, RestSharp is a popular and convenient open-source HTTP client library for .NET. It’s easy to install and add it to your C# project. RestSharp parses both XML and JSON formats.
RestSharp advantages. RestSharp’s functionality allows for straightforward test creation, serialization and deserialization. It can handle synchronous and asynchronous requests with a wide list of ready-made authenticators. Uploading files and forms in multiple parts, RestSharp cuts down on upload times.
EPAM’s Oleksandr Fomenko highlights its powerful features and considers RestSharp the best solution for processing HTTP requests. He also accents its clean implementation: “ RestSharp allows you to easily implement any type of HTTP client.”
SoapUI: a functional testing tool for both REST and SOAP APIs
SoapUI is an API testing tool in SmartBear’s ReadyAPI suite. They also have LoadUI for performance testing. We have already described another SmartBear product used for UI testing — TestComplete — in our article on automated testing tools.
SoapUI pricing plans. You can go for an open-source version that allows for:
- creating custom code (using Groovy) and complex scenarios;
- testing asynchronously;
- mimicking API before it’s implemented.
SoapUI Pro includes such advanced features as an assertion wizard for XPath, form editor, and SQL query builder. It costs a pretty penny — for individual users $599/year and for CI teams — $4,190/year, including Docker Support. There’s also a customizable plan for API-first teams.
Katalon Studio: automation tool with inbuilt API testing mode
Designed primary for UI testing, Katalon is also an integrated environment meant to easily generate and execute API tests.
Katalon advantages. Its edge over other automation tools like Selenium or Ranorex is that it doesn’t need additional integrations to test APIs. Katalon has an inbuilt API testing mode with the following capabilities:
- code auto-completion, inspection, and snippeting;
- references and debugger;
- inbuilt response viewer with auto-formatting;
- search for access to artifacts;
- tests importing from API testing and editing tools (e.g. Swagger, Postman, WSDL).
With Katalon, you can test all types of REST, SOAP/1.1 and SOAP/1.2 requests and multiple data sources.
Katalon pricing plans. There’s a free version but it’s not open-source. Dedicated consulting services come with Premium Support ($5,000/year) and Solution Support (starting at $30,000).
If Katalon has piqued your curiousity, we have an article with an objective review of this software.
API testing framework vs ready-made API testing tool
Choosing among the tools we described, keep in mind that some of them are better for writing an automation framework (RestSharp, REST Assured, and Requests), while other tools (Postman, SoapUI, Katalon, and jMeter) can be utilized as they are.
So, which path to take?
Writing your own framework gives you more freedom as you’re not limited to the capabilities of a tool and its existing plugins. You can customize it your way: Add any library suitable for the chosen coding platform, introduce complicated logic in your tests, build a reporting standard convenient to you. You also get more flexibility in managing your test data. But you enjoy these perks only if you possess strong coding skills.
Otherwise, you’d better look towards a ready-made tool. Its user-friendly interface and low-threshold coding requirements will be a relief for you. The pitfall here is that they usually address general API testing issues. But the good thing is that their community can easily help you solve your more-or-less common problems.
To sum up, a ready-made tool like Postman is good for starters. Developing your own automation framework is a more advanced level, but it’s also rewarding in terms of functionality you can add to your testing solution.
Originally published at AltexSoft tech blog “API Testing Approaches and Tools: Postman, Rest Assured, jMeter, and more”