Jaspar Casey - 20 April 2021
When software doesn’t work properly, the effects range from the merely inconvenient to the potentially catastrophic. Thankfully, there are many steps we can take in order to avert disaster and keep everything rolling smoothly along. One such area is performance testing, which refers to the broad set of measures taken to determine whether software meets various speed, stability, and scalability requirements. An area of performance testing of particular interest is load testing.
What is load testing?
Load testing is the subset of performance testing that looks at a system’s ability to perform under high user loads. According to Joe Colantonio, this is done “to see if a system’s infrastructure can handle the load without compromising functionality or performance.” As workload increases, load testing helps us understand the performance of the system and those related systems. When done correctly, load testing allows teams to verify an application’s behavior under both normal and peak load conditions.
Examples of load testing
The nature of your load test depends on what’s important to you. If you’re a file sharing service, you’ll need to make sure that your systems can handle a large number of users downloading and uploading files concurrently. If you are an IT administrator, you might want to make sure that your company email service can handle large volumes of messages that are delivered in batches, such as internal newsletters that may have rich media within them. If you’re an ecommerce site, you’ll want to be prepared for large numbers of customers at certain times, such as on Black Friday. A use case we’ve been involved with recently is load testing of the various systems hospitals rely on to roll out COVID-19 vaccines. The point is, whatever your business or are of concern, load testing should be a consideration.
Benefits of load testing
Load testing is helpful for identifying bottlenecks in your system that emerge under various workloads. As the load is gradually increased, you should observe how various components respond, such as databases, servers, and client-side devices (usually web browsers).
This last group is particularly important, as in many contexts, the effects on the end user experience will be most relevant. After all, an overloaded server is a headache for an engineer; but an overloaded server that causes an entire application or website to not function has a wider impact. Failure to adequately prepare for peak user loads could have devastating effects on your business.
There are a number of other benefits to load testing outside of averting disaster. According to Microsoft, load testing helps:
- Detect functionality errors
- Determine how much load the hardware can handle before limits are exceeded
- Collect data for capacity and scalability planning
- Determine the adequacy of the hardware environment
Load testing vs stress testing
While both look at how applications and systems hold up under heavy loads, stress testing in particular looks at extreme cases. The idea here is to identify the breaking point, the point at which there are simply too many users making too many requests. Stress testing uses conditions outside of normal working conditions. While load tests seek to help teams understand the performance of their systems, stress tests uncover the stability and robustness of those systems. However, some stakeholders may diminish the importance of stress tests, because the scenarios they include are by design unrealistic.
How to get started with load testing
There are a wide range of tools and approaches that organizations can use for load testing. These typically fall into four categories: vendor-backed enterprise load testing tools; open-source tools; custom, in-house tools; managed load testing conducted by a third party.
LoadRunner, which has been around since 1993, and JMeter, which emerged as an open-source alternative, are widely used. Some organizations choose to develop their own load testing tools in-house, though they often base these around open-source components. A separate option is outsourcing your load testing through a managed load testing service, where a team of dedicated experts can do the load testing for you.
When planning your load testing approach, make sure that you are always thinking of the underlying question: how does the performance of my systems affect the end user experience? This will ultimately help you to create realistic and useful load tests.
It’s also vital to be able to actually measure the impact on each component and the end user experience of peak load conditions. Traditionally, the metrics center around response time. For an end user, a slow response could manifest itself in a page that takes a few seconds longer than normal to load; or it could mean that vital components don’t load at all, causing a frustrating or confusing user experience.
Looking at response time, it first helps to understand if the desired response time is:
- Already measured — do we have a baseline to compare our load test results to?
- Relevant to business processes or logic
- Realistic — should we expect our ecommerce site to load as fast as Amazon?
- Achievable — do we have the hardware and other resources to consistently reach our desired response time?
- Measurable — have we got the proper tools in place to accurately measure response time? Can we also measure other impacts on the user experience besides load time?
Load testing best practices
Many of the best practices for load testing overlap with best practices for software development. One such concept is the idea of using reusable components for load testing. These could be common scripts, pre-built models, or snippets of code that help your teams easily assemble load tests.
Some other best practices and questions to keep in mind:
- Are the goals for this test relevant, realistic, achievable, and measurable?
- Have we identified all of the parameters that we will need to measure? These can include but are not limited to client-side response time, server memory used, page faults/second, hit ratios, throughput, etc.
- Do we have the ability to monitor and capture performance data in an accurate way that won’t slow down our systems?
- Are we testing the live production system? If not, what configuration are we testing, and what are the specifications?
- Do we have a clear sense of the most relevant user journeys that we should be testing?
- Will an HTTP-level test be sufficient, or do we want our virtual users to interact with the graphical user interface (GUI)?
Because load testing ultimately supports your end users’ ability to use your application, it’s important to understand how they are using your application. When planning a load test, you need to make sure that the conditions you are imposing on your systems reflect how a high throughput of users would actually interact with your application. This means that you need to first accurately map critical user journeys that you would expect to be taken in peak load conditions. For example, if you’re an ecommerce site and you are about to release a highly anticipated product that has been heavily promoted on social media, you’ll need to understand the most typical user journeys your customers will take in order to buy this item. From here, you’ll want to repeatedly test these user journeys and see how they are impacted by increasing loads.
Common load testing challenges
Engineers face a number of challenges when conducting load tests. Besides the considerations of measurability listed above, teams must also consider the technical skills needed to use load testing tools, as well as the price of using these tools. However, assuming those things are accounted for, the most important challenge that you’ll face is that your load tests reflect real-world conditions that your systems are likely to experience. Here are a few other things to consider:
Measurability through monitoring
Due to the range of systems and components that need to be measured, it is important to have the ability to accurately monitor and report your key metrics. A key part of this is instrumenting your application or site. This allows you to understand performance in various circumstances, including what peak loads actually look like. Synthetic monitoring tools can be used to check for performance issues at regular intervals. Alerts can then be sent to various stakeholders when certain triggers are hit. Modern solutions like Eggplant Monitoring Insights allow teams to take advantage of the latest advances in monitoring technology, such as configurable alerts sent via Webhooks.
Team members conducting load tests will benefit greatly from prior experience. There are many factors that affect a site or application’s ability to handle peak user load, so the more experience and expertise your team has, the better. Of course, even experts will run into issues, so it’s important to learn from mistakes and keep seeking learning opportunities. An example can be found in the story below about lessons learned by Moonpig’s performance team. It can also help to listen to performance and load testing podcasts.
Having adequate load generator readiness
Load generators are machines that run your virtual users through your system under test. To make sure these computers can reliably do their job, you’ll need to make sure there are enough machines, they don’t have any extraneous software, they are properly connected to the relevant network, and you give them sufficient downtime.
Understanding the difference between GUI and HTTP virtual users
Most load tests involve scripts interacting with a site at the HTTP level. While this is an efficient way to run a load test, you may miss out on some of the insights possible when testing at the GUI level. Remember, GUI virtual users are much more resource-intensive, so this should factor into your plan.
Timing of load tests
You’ll likely have an idea of when your site or application experiences the highest loads. It’s therefore important to consider the timing of your load tests. You need to strike a balance between running the tests far enough in advance to be able to make necessary adjustments to prepare for real traffic, but close enough to the event that your application reflects what real users will see.
Tests should reflect real-world conditions
When creating a load test, it’s important to reflect the real-world conditions that your systems are likely to experience. One key part of this is to build in ‘think time’ into your logic. What this means is that your virtual users will need to pause between steps to replicate a real user thinking. After all, virtual users can zip through a workflow much faster than real people can. Therefore, to faithfully recreate peak load conditions that your systems will see in real life, your load tests need to account for the speed at which real people will use your application.
Moonpig, a greeting card and gift company, has fairly predictable spikes in activity around Valentine’s Day, Mother’s Day, Father’s Day, and in the leadup to Christmas. For one of their load tests, they had a script to repetitively carry out a user journey to purchase a greeting card. Every 15 seconds, the user would make the same order. They noticed an odd phenomenon, in which their order rate wasn’t increasing with the increased traffic. Initially, they thought this meant the system was dynamically adjusting itself to the increased load. However, on closer inspection, they realized that what was really happening was that as load increased, the site would slow down, which would slow down these virtual user journeys. Journeys that would take 15 seconds would now take 25-30 seconds.
In other words, the load test was simultaneously slowing the system and slowing the load test itself, giving the impression that the user experience was unaffected. Instead, Moonpig’s engineers used a feature on JMeter to select an arrival rate. This meant that as system load increased and things started to slow down, the number of concurrent users increased. Of course, this is what would be expected under real-world conditions. Therefore, it’s vital that whatever load testing setup you have, you are able to faithfully replicate real-world conditions so that you can anticipate how your systems will cope with this increased load.
Key points to consider when selecting a load testing tool
The main decision to make when it comes to selecting a load testing tool is whether you want to go open-source or enterprise-grade. Open-source tools have many advantages, often boasting active communities, libraries of helpful scripts, and of course, they are free to use. However, there are hidden costs, such as the lack of official technical support, the specialist skills required to utilize them, and their complexity for those engineers who may lack a lot of experience with that particular tool.
Commitment of resources
On the enterprise side, the price tag can be a challenge to carefully weigh up. However, given the potential consequences of systems that can’t stand up to peak loads, this is a price many IT leaders are willing to pay. Therefore, when embarking on a load testing journey, make sure that leaders are willing to commit appropriate resources. Otherwise, you’ll run the risk of an inadequate load testing program that suffers from a lack of commitment.
Monitoring and diagnostics
When creating a load testing plan, it’s important that you have the necessary tools in place to effectively monitor and diagnose performance issues. Make sure your load testing tool has these capabilities or at least integrates well with tools that do.
Because load testing should be part of your CI/CD pipeline, it only makes sense for your load testing tool to integrate with platforms like Jenkins and Hudson. Another key feature to look out for is the ability to communicate with systems via open APIs.
Compatibility with operating systems and devices
Depending on your application or site, your load tests will need to account for users accessing your systems across different types of devices and operating systems. Some systems have added complexity, such as desktop virtualization software like Citrix. You’ll need to make sure that your load testing tool can handle these environments.
Hopefully, you’ll now have a better understanding of what load testing is and how important it can be to your business. In our hyper-connected world, we are increasingly reliant on software to do just about anything. When these applications don’t stand up to peak usage, alarms go off and users get frustrated, or even worse: they give up and go elsewhere.
Understanding the need for load testing is one thing, but knowing how to build and execute a plan that upholds business goals is a bit trickier. Eggplant offers both Managed Load Testing as a service, as well as our popular Eggplant Performance product, which is used by AutoTrader, British Heart Foundation, and numerous other organizations. Eggplant Performance can seamlessly integrate with your CI/CD pipeline, and via open APIs, can easily connect with open-source tools like JMeter. For more information, take a look at our site or get in touch with a member of our team.