Purpose of this chapter
This chapter will focus on preparing your tests in order to run smoothly.
- Why objectives are essential:: First, we will talk about Objectives. We will also look at a few examples and guidelines to define good objectives,
- Skills and resources: After this we will look into the required skillset and who can provide it. That way you can schedule time slots with experts during your campaign,
- Selecting the test environment: Then we will have a few words on how to select the right test environment. We will take some time to look at the pros and cons of each environment,
- What is a test plan: We will talk about the importance of a test plan. Whether you do one or not is up to you. But it is important to know what it stands for while making this decision.
We have talked about the importance objectives earlier. Let’s take more time. Every objective you define must provide the means to answer a simple question: Why do we test? As we discussed on risk based testing, this simple question should answer to a performance risk.
With this in mind, here is a list of objectives examples along with the risks associated:
|Risks covered||User satisfaction||Infrastructure issue||Stability||Brand image|
|"To have a response time under XX seconds"|
|"Load balancing or failover works fine"|
|"Error rate is under XX%"|
|"The infrastructure can handle XXX users"|
What is important is to have at least one clear goal. But most campaign actually have several objectives. Some might not be clearly identified from the start. But that’s why the preparation phase is important.
Generally you want to avoid testing only for the sake of testing. Defining objectives will help understand the expectations. It gives an occasion to reflect on the reasons behind the test campaign. You want to detect any kind of hidden agenda behind the tests. Because if there is one and you are not aware of it, the campaign is likely to fail.
Now on the question of why objectives are so important it can help to look at a few situations to realize it.
Selecting the best test scenarios
Say your goal is to check that 1000 users can be logged in at the same time. There’s no need to record a whole user journey through your application. Which means you can rely on simpler test scenarios and focus your effort elsewhere.
Another example: Your goal is to check the response time of the checkout page.
First your test scenario must include it. Then you need to consider the different ways someone can submit this page. Some might be slower than others, in particular if you consider third parties. Also to do that you must make sure you are logged in at this point. Which means thoroughly checking for errors after the login.
Monitor what matters the most
Pretty much the same here, you want to focus time and effort to what matters the most to your test.
What goes well without saying, goes even better when you say it, so:
- When the load balancing is a risk, make sure to monitor the load balancer,
- When there is an auto scaling mechanism, make sure you have a way to monitor the new servers,
- Finally, you might think you do not need to monitor the failover database. But when the main one fails you’ll be glad you didn’t. Also measuring the impact of data replication on an inactive database is always interesting.
Choose the right environment
We will discuss this one later in great length. But selecting the right environment to test is difficult. Here again your objectives will help. Some of the items you have to test might not be there in pre-production. Or they might be impossible to test in production.
Select the tests to launch
The types of test are tied to your objectives. Let’s compare regular load and big sales on an e-shop.
For the first one, you want to run the most average test scenario. Meaning you want to use the stats to run a scenario that will reproduce what an average user does. You will probably not go through the purchasing tunnel for every user. You want to keep it running long enough so that you get all relevant metrics.
For big sales, what matters is to make sure the applciation handles the initial load. So you want a short test scenario that will go right away to a sale. What matters is how many orders you can handle concurrently. And you will probably run it only for 15-30 minutes. Because after the opening the load goes down quickly.
And of course that’s just an example. Here are a few others:
- Internal applications might have a longer peak of activity every day,
- Institutional websites will have sustained peaks after marketing campaigns,
- Registration applications tend to have a peak on the last day of registration.
With the right objectives in mind selecting the right test is very easy.
Refer to objective when analysing
Judging the performance is only achievable in regards to your goals, otherwise you can’t be objective. And yes of course pure objectivity is impossible. Still that does not mean you should not try. Sentences like “performance is good” have no value to your reader. The acceptable performance is different for every application, transaction, etc… It must be good in regards to the expectations, or as defined in the objectives.
Know when the tuning ends
You could continue optimizing the application endlessly. There objectives help you call it a day. That does not mean you can’t try to go further if time remains, but at least you know when to stop.
Also the cost of optimization tends to get higher when most obvious defects are already fixed. It is important to know if these costly optimizations are required at all. You could schedule them for a later version instead.
The Pareto Principle is often true when dealing with optimizations: 20% of the tuning usually results in 80% performance improvement.
Baseline / SLA
Of course sometimes it can be difficult to identify a clear objective. We won’t be able to cover every possibility. But here is a diagram to help you in some situations.
First in case the application is not new, you might already have statistics. Or at least you can look at resource consumption or logs on the servers. This greatly helps understand what real users do on the application. And it can also point you to unusual peaks of traffic. It also tells you that you will have to focus on non regression testing. In particular if the new version is supposed to fix a known issue.
It gets more complicated when the application is brand new. Of course if any SLAs exist, you should use them. But that’s not always the case. A first step could be to look for any outstanding technology or framework (for instance first time with varnish). These can take effort to use or configure properly. Thus you are more likely to find issues with them. You can also run some capacity planning tests. This way scaling up these components will be easier in the future.
Another thing to consider is if the application is public facing or not. A public facing application means user expectations are standard. This means you can apply standard modern SLAs like response time under 1s. Or run longer tests to check for 24⁄7 coverage.
Try to find out what kind of objectives a test campaign would have on the following application:
Hint: Each sentence matches at least one risk.