Objectives - Load Testing Preparations

Objectives - Load Testing Preparations

Methodology 7 minutes

A chapter about load testing Objectives. Why they are so important to cover risks. And also how to define proper objectives.

Purpose of this chapter

Purpose of this chapter

This chapter will focus on preparing your tests in order to run smoothly.

  • Why objectives are essential:: First, we will talk about Objectives. We will also look at a few examples and guidelines to define good objectives,
  • Skills and resources: After this we will look into the required skillset and who can provide it. That way you can schedule time slots with experts during your campaign,
  • Selecting the test environment: Then we will have a few words on how to select the right test environment. We will take some time to look at the pros and cons of each environment,
  • What is a test plan: We will talk about the importance of a test plan. Whether you do one or not is up to you. But it is important to know what it stands for while making this decision.

Objectives

Objectives

We have talked about the importance objectives earlier. Let’s take more time. Every objective you define must provide the means to answer a simple question: Why do we test? As we discussed on risk based testing, this simple question should answer to a performance risk.

With this in mind, here is a list of objectives examples along with the risks associated:

Risks covered User satisfaction Infrastructure issue Stability Brand image
"To have a response time under XX seconds"
"Load balancing or failover works fine"
"Error rate is under XX%"
"The infrastructure can handle XXX users"

What is important is to have at least one clear goal. But most campaign actually have several objectives. Some might not be clearly identified from the start. But that’s why the preparation phase is important.

Generally you want to avoid testing only for the sake of testing. Defining objectives will help understand the expectations. It gives an occasion to reflect on the reasons behind the test campaign. You want to detect any kind of hidden agenda behind the tests. Because if there is one and you are not aware of it, the campaign is likely to fail.

Why objectives

Now on the question of why objectives are so important it can help to look at a few situations to realize it.

Why objectives

Selecting the best test scenarios

Say your goal is to check that 1000 users can be logged in at the same time. There’s no need to record a whole user journey through your application. Which means you can rely on simpler test scenarios and focus your effort elsewhere.

Another example: Your goal is to check the response time of the checkout page.

First your test scenario must include it. Then you need to consider the different ways someone can submit this page. Some might be slower than others, in particular if you consider third parties. Also to do that you must make sure you are logged in at this point. Which means thoroughly checking for errors after the login.

Monitor what matters the most

Pretty much the same here, you want to focus time and effort to what matters the most to your test.

What goes well without saying, goes even better when you say it, so:

  • When the load balancing is a risk, make sure to monitor the load balancer,
  • When there is an auto scaling mechanism, make sure you have a way to monitor the new servers,
  • Finally, you might think you do not need to monitor the failover database. But when the main one fails you’ll be glad you didn’t. Also measuring the impact of data replication on an inactive database is always interesting.

Choose the right environment

We will discuss this one later in great length. But selecting the right environment to test is difficult. Here again your objectives will help. Some of the items you have to test might not be there in pre-production. Or they might be impossible to test in production.

Select the tests to launch

The types of test are tied to your objectives. Let’s compare regular load and big sales on an e-shop.

For the first one, you want to run the most average test scenario. Meaning you want to use the stats to run a scenario that will reproduce what an average user does. You will probably not go through the purchasing tunnel for every user. You want to keep it running long enough so that you get all relevant metrics.

For big sales, what matters is to make sure the applciation handles the initial load. So you want a short test scenario that will go right away to a sale. What matters is how many orders you can handle concurrently. And you will probably run it only for 15-30 minutes. Because after the opening the load goes down quickly.

And of course that’s just an example. Here are a few others:

  • Internal applications might have a longer peak of activity every day,
  • Institutional websites will have sustained peaks after marketing campaigns,
  • Registration applications tend to have a peak on the last day of registration.

With the right objectives in mind selecting the right test is very easy.

Refer to objective when analysing

Judging the performance is only achievable in regards to your goals, otherwise you can’t be objective. And yes of course pure objectivity is impossible. Still that does not mean you should not try. Sentences like “performance is good” have no value to your reader. The acceptable performance is different for every application, transaction, etc… It must be good in regards to the expectations, or as defined in the objectives.

Know when the tuning ends

You could continue optimizing the application endlessly. There objectives help you call it a day. That does not mean you can’t try to go further if time remains, but at least you know when to stop.

Also the cost of optimization tends to get higher when most obvious defects are already fixed. It is important to know if these costly optimizations are required at all. You could schedule them for a later version instead.

The Pareto Principle is often true when dealing with optimizations: 20% of the tuning usually results in 80% performance improvement.

Baseline / SLA

How to

Of course sometimes it can be difficult to identify a clear objective. We won’t be able to cover every possibility. But here is a diagram to help you in some situations.

Existing application

First in case the application is not new, you might already have statistics. Or at least you can look at resource consumption or logs on the servers. This greatly helps understand what real users do on the application. And it can also point you to unusual peaks of traffic. It also tells you that you will have to focus on non regression testing. In particular if the new version is supposed to fix a known issue.

New application

It gets more complicated when the application is brand new. Of course if any SLAs exist, you should use them. But that’s not always the case. A first step could be to look for any outstanding technology or framework (for instance first time with varnish). These can take effort to use or configure properly. Thus you are more likely to find issues with them. You can also run some capacity planning tests. This way scaling up these components will be easier in the future.

Public applications

Another thing to consider is if the application is public facing or not. A public facing application means user expectations are standard. This means you can apply standard modern SLAs like response time under 1s. Or run longer tests to check for 247 coverage.

Exercise

Try to find out what kind of objectives a test campaign would have on the following application:

Exercise

Hint: Each sentence matches at least one risk.

Solution