Load Testing Definitions - Methodology Introduction

Load Testing Definitions - Methodology Introduction

Methodology 5 minutes

Introduction to our course on Load Testing Methodology: the agenda, an overview of the main concepts, and load test definitions.

Welcome to the first chapter of our load testing methodology training course. In this course we will cover all the concepts required to perform meaningful load tests. This first chapter is about the introduction. We will discuss the agenda and have an overview of the main concepts. Each concept will later have its own section in the next chapters.

Purpose of this training

Purpose of this training

As you probably figured out by now, this training focuses on methodology. Although some screen captures come from existing tools. The content in this course is tool agnostic. Having said that, we will focus on protocol based testing as performed in tools like JMeter. We will take more time to discuss this choice later on. In particular when comparing the different ways to run a meaningful load test.

If you are looking for guidance on OctoPerf try our tutorials and we also have a lot of blog posts regarding JMeter.

Definitions

Definitions First it is good to agree on some vocabulary. It is important that we speak the same language as we will use these concepts all along the training.

  • Test scenario or User journey: List of actions or screens a real user would go through on the application.

  • Test script or Virtual user: Program that reproduces the test scenario. Note that virtual users can refer to all the concurrent users running during a test.

  • Runtime scenario or Load policy: Configuration of a load test, including: concurrent users, ramp up, duration, etc.

You might already be using some of these, or even other words to express such concepts. But once again the purpose of this section is to make sure we understand each other.

Load Testing definition

Load Testing definition

Before going further, we should agree on a definition of load testing. For a definition, Wikipedia will do just fine. The key appears to be that load testing is a mix of putting demand on a software and measuring its response. Both steps are important for a successful load test. We will see why and how these topics can be addressed all along this training.

But let’s also take some time to talk about about common types of load tests. Here again the purpose is to ensure that we use a common vocabulary over the course of this training. If you would rather nickname your tests any other way, go wild.

Performance test

This usually refers to testing the expected load on an application. It is usually one of the first tests you will run on any given campaign. You should use it as a benchmark for further tests.

Volume test

The goal here is to simulate database growth. This is likely to happen over the first months or years. It is important to predict how the application responds to this. Of course it helps sizing the database and also tells you how much time you have before it gets too large.

Endurance/soak test

This test reproduces a whole day of production. Which by the way might not be a day long depending on the application. It ranges from 8 to 48 hours depending on your user base and application’s purpose. You need to run this kind of test to spot memory leaks and network shortages. It is usually overlooked since it takes time to prepare and run. And of course any re-run might be difficult to organize for the same reasons. Note that when testing a production environment, it might not even be possible to find a time slot to run it. I would still recommend to run one when possible because:

  • Memory issues might not appear for the first few hours of testing. They come from inefficient/incorrect garbage collection or code issues. They would be hard to spot on a short test.
  • Background tasks that run at a specific time every day. These could also have an impact on performance. Unless the test is running at that specific time you would not see them.
  • Logging configuration. Incorrect logging configuration may cause disks issues when running out of space. Only a longer test might allow to spot these.
  • Mutualized components nearing their limit every day. As with background tasks you would not see this issue if the test is not running at the time it happens.

In short, keep in mind this test is essential to figure out how the application performs in the long run.

Stress test

As its name indicates, you want to put a lot of stress on the application. And not only because testers like breaking every application. The goal is to prepare for a real peak of activity. This means much more stress than the expected load, it could also be on a much shorter period of time (flash sales, etc..). You should not run this test first since simpler tests might yield better results. At least better results in regards to the time and money investment. Aiming too high on your first test is a common mistake.

Scalability test

The name pretty much spoils the surprise. This test will help scale the application’s infrastructure. Several ways to do it. First you could run a stress test on a single group of servers and assess how much load each server can handle. This should give you an idea of how many servers you need.

You could then test the auto scaling. See if it starts or stops machines as quickly as required, or just test the whole platform at the desired load.

Note that these few tests are among the most common ones but of course you could add many more tests to this list:

  • Unit test: Run each script a few times to check them and get best response times for benchmarking purpose,
  • Failover tests: Stop/start one of the servers during the tests and measure the impact.