Load Testing Process - Methodology Introduction

Load Testing Process - Methodology Introduction

Methodology 7 minutes

First overview of a complete load testing process: preparation (load test qualification sheet), design, execution, analysis and reporting.

Load test preparations

Load test preparations

The very first step of any project is to gather the requirements. This is the main purpose of the qualification sheet. We are not just talking about a basic list of questions here. As load testing can be an unfamiliar process to many, it should also be used as a guide. The idea is to ask the right questions AND explain why you are asking them.

A good example is the expected load level for the tests. To emulate realistic user interaction on any application you must know:

  • The proper test scenarios: To ensure the test emulates a real life situation.
  • The amount of concurrent users to simulate: Because you must test pools and queues with the right amount of load.
  • The activity over a period of time, for example XX purchases per hour, XX product pages viewed per day, etc. This will help pace the execution of your test scenarios.

It is probably not clear to a non-tester why you would need all this for a “simple load test”. That is where the qualification sheet can help you. First by explaining that you can only express load as actions per unit of time. Whatever the action and unit of time. And that reproducing the load of a real life situation is not enough. You also have to consider the concurrent user sessions.

Let’s take an example. Say you can reproduce the load through 100 users. These users will never stop navigating through the application. Which makes them 10 times more active than a real one. But in reality there are 1000 expected users on your application. Testing with 100 very active users does not ensure your servers can handle 1000 sessions. Maybe the threads on your front server or database connections cannot do more than 100. If you are a tester, or if you are familiar with inner mechanism of a web/app server or database it might be clear to you. But it is probably not to everyone.

It is only by advocating the reasons why you need this information that you can get people on board. And getting them on board is essential to gather all prerequisites.

Now about the content of this sheet. As usual do as the context commands but I would always ask at least the following:

  • Objectives,
  • Test scenarios,
  • Load policy,
  • Infrastructure details,
  • Application technology,
  • Security and timings.

It is also the perfect occasion for you to get specific info. For instance:

  • Underlying frameworks used in the application,
  • Security measures that might need temporary deactivation.

It will help later with sizing the campaign (work required, delays and infrastructure).

And also putting all this together will allow you to create a proper test plan later on. Most often than not some sections of this sheet remain empty. Because people don’t know what you need or don’t have an answer. But mentioning it early on will get them thinking about it. And when you ask again later you will have an easier time getting an answer.

Added bonus: A detailed qualification sheet will prove you have a structured approach.

Load testing process

Load testing process

Before wrapping up the introduction, lets walk through a typical load testing process. This will allow us to cover the headlines of the next sections of this training.

Preparation

That is usually where everyone meets and discuss the purpose of the tests. The most important step here is to define the objectives. There is nothing less effective than a test campaign without a clear purpose in mind. Then you want to consider writing a test plan. It is a good way to write down everything related to the test and avoid later disagreements. It will also serve as documentation for the next test iteration. Which can be very valuable when someone else runs the tests.

Also you will list all prerequisites and ask the different contributors to put them in place. To that end, writing a dedicated section of the test plan is fine. But keep in mind that most people will not read it. It is better to list them in an e-mail and put them in plain view. Even better, list them during a call or meeting. But even this way do not assume everything will be ready in due time. Most contributors will have a lot of tasks to perform and yours might not be a priority. You must follow up to make sure the prerequisites’ importance is well understood. It is the only way to know for sure that they are being implemented as expected.

For instance, here is a quick list of prerequisites. You should ask as soon as possible in the discussion since they might take time to setup:

  • Datasets creation: unique login/password list, etc.,
  • For test scripts stability, application version must remain the same during the tests,
  • A test architecture dedicated to the tests. The load tests will impact other operations such as functional tests,
  • Preparation of the reset procedure between two tests. (database restore, etc.).

Design

In here we regroup everything related to preparing the load tests:

  • Creating test scripts from agreed upon test scenarios,
  • Use (test) scripts to prepare a set of data (logins, references, etc.),
  • Installation of the test environment,
  • Prepare load policy based on information from the test plan.

Runtime

Once all preparations are in order, you can launch all your tests. Each one of them will emulate: * a different way to use the application * a different precondition (more or less servers). Important thing to note: the importance of the Unit test. This should be the very first test you will run for any test campaign. And you should use it as a benchmark for all further tests, we’ll have a more detailed look at this later in the training.

Analysis

Although related to runtime we’ll reserve an entire section to result analysis. Mostly this relates to:

  • Why average values are not always enough to ensure proper analysis,
  • How comparison to a benchmark can help spot the issues,
  • Validating performance through reproducible tests.

Reporting

Here again we could merge this with analysis. But reporting the results of a testing campaign is already a big challenge. We’ll see how you can organize results in a more efficient way. And what it means to present them to different kind of profiles.

Challenges exercise

To test your knowledge before the end of this chapter, re-arrange the following challenges in the proper category (preparations/design/analysis):

Challenges exercise

Try to refrain from checking the solution too early. A good exercise will help you memorize the concepts.

Challenges solution

Challenges solution

Wrap up

Wrap up

We’ve been through many concepts during this introduction. It is always good to sum up the concepts before going to the next subject.

First the definitions, because speaking the same language is important. For instance the difference between test scenario and test script is not easy to grasp at first. Also important to consider: Load testing is a combination of generating load and monitoring the results. Both are important to get good results. And common load tests names help understand the purpose of each test quickly.

Then we also discussed why load testing is important. In short it helps provide better brand image and better SEO rankings. While keeping costs under control.

The next matter was how to load test. Tests must be realistic and reproduce real life traffic. Because you want to stress all layers of your infrastructure. Also run risk based tests to test what matters most. Having this in mind will optimize the testing process. Then we discussed testing as early as possible. Having in mind that defects spotted this way are cheaper to fix. But that a full blown load test before release is still mandatory. We also mentioned CI integration as a good way to ensure a better automation. Of course here as well you must consider the ratio between outcomes and efforts.

Last but not least we’ve been through the load testing process. The qualification sheet is the first step. It is a key to gather information and educate on the load testing prerequisites.

The rest of the process involves several steps:

  1. The preparation phase to narrow the scope around clear objectives,
  2. The design phase to prepare test scripts using varied datasets,
  3. The runtime phase to run tests related to each goal,
  4. And the analysis phase to compare results to benchmark and prepare a report.

This wrap up ends the introduction. Next chapter will be centered around the preparation phase.