Why Load Testing
Now that we have defined what a load test is, let’s focus on why we would need to run one. Here are a few of the reasons why you would run a test. Of course some of the statistics might seem outdated or provided by a non neutral source. But we are more interested in the concepts behind them that remain valid nonetheless.
Preserve brand image
First you do no want customers to remember your brand as “this website that went down last year”. Yahoo and security leaks of account information is the perfect example of what not to do in that area. It happened so many times that they have their own wikipedia page just about it … #securityshaming.
Secure marketing investments
Preserving your brand image is one thing. But you also want to make sure you make the most out of your marketing investments. To do that, your application must be able to handle the load after featuring on tv or a popular website. But also mind that sustained marketing bring sustained growth of your user base. And in this case running tests twice a year might not be enough to spot the new bottleneck of your platform. Running regular tests, if possible automated, would be much more efficient.
Better conversion rate
A faster website will bring you more customers. A faster internal application will help users be more productive. Improving performance will always pay in the long run. Also mobile users tend to expect faster performance than desktop users. Even though we all know network conditions are difficult to predict. This only makes performance even more important to user satisfaction.
Many say that hardware is becoming a commodity. Without going in too many details, it is easy to see that hardware is much more affordable these days. Because of this adding a server might seem cheaper than optimizing the code or fixing the real issue. But keep in mind that servers are usually rented and not owned. And even when you own them, maintaining a large fleet of machines comes with a lot of hidden costs. So in the long run it is usually cheaper to optimize your application. Because stacking issues on top of each others will only get more and more expensive. And of course if your user base grows on a poorly optimized application, costs will skyrocket.
Improve SEO rankings
The better your search engine ranking, the more visibility you get. That is basic math. The real question is how to achieve that. Although SEO sometimes seems like black magic, some basic facts remain true. For instance, response time plays a huge role in your SEO ranking. Google tends to rate a website as slow starting from 1.5 seconds. And the good news is that you can optimize it through load testing.
To sum up, I like to use this well known quote:
How to Load Test
Now that we have walked through what and why, let’s see how to perform load tests. This is where we will spend the most time. We will start looking at this through 3 angles that we will develop throughout this training:
- Realistic: If your tests stray too far from what real users do, you will not find real bottlenecks. Worst, you might get false positives and spend time/money on non-issues. Running realistic tests is the main focus of this training.
- Risk based test: You do not need 100% test coverage. Not even close. Achieving that would be a waste of time and money. Instead, you should identify the risks and provide a proportionate answer to each one. This step is very tricky since there is no definitive answer on how to do this. And we all tend to add too many things to our tests from time to time. We will focus on a few metrics that can help.
- Test early: The cost to fix a defect will only grow as your application gets closer to its release. You want to find performance defects ASAP. But it is easier said than done. Our goal is to make some of the former become the latter.
Realistic means that your test should reproduce the behavior of real users.
You may think you can achieve that by mass calling a URL on your website. I call this hammering a URL. It works up to a point but soon your front servers will serve information from their cache. So in the long run such a test will not stress your backend. If you go on with hammering, it might even lead you to optimize the caching policy in a wrong way.
Another way would be to replay production traffic on the network level. If you can get your hands on production logs or network components that record the traffic. Then you can reproduce said traffic exactly as it was. This should be very realistic, right? Well I hate to break it to you but the answer is no. Do not get me wrong, such a test is very good to check the network. Which makes sense as it is its purpose. But on the servers side you are replaying outdated sessions. That will generate a lot of errors on your backend. Errors have a different resource cost than a normal session. So in the end it will lead to incorrect conclusions.
Instead you want to stress all layers of your application. To do that you must create traffic that will generate proper load on the back end and database servers as well.
To do that, you want to reproduce either:
- Real device tests using a real browser or real mobile devices,
- Protocol based tests sending requests with variable contents, reproducing real user sessions.
We will discuss both options, their pros and cons in a next chapter.
Risk based testing
Risk based testing is a fancy way of saying that you should test only what matters most. This means that applying the same set of tests to any application is not the most efficient way to go. As we have seen before, each test has a purpose and they do not apply to every application. You have to adapt to the context each time.
On top of this, we can say that not all functional test scenarios will be interesting for load testing. You want to optimize the risk coverage with as few test scenarios as possible. Performance test scenarios can be heavy on design and maintenance. So having an unrealistic amount of them will put a lot of pressure on the preparation phase.
It can also lead you in an infinite design loop. For instance if the application evolves faster than you can update your test scripts. A rule of thumb is usually to say that you can cover 80% of the risk through 20% of the test cases. We will take more time to cover this in the design section. Note that a tool efficient on the design phase helps increase your coverage. But it is not the topic of this training.