With the rise of Devops, developers are more likely to run the tests. The idea is that the same people will develop, test and operate the application. In combination with a lot of automation this allows shorter delivery cycles.
It is a good solution when considering the technical skills required for load testing. Understanding the way the application communicates is much easier for a developer. But there are also some issues with this.
Can a developer be objective about the quality of his own code?
When the same person develops and tests, it is difficult to remain objective. Thus he may spot fewer issues. Also a developer might not know the exact purpose of the application. Or at least he will have fewer contacts with end users than some other profiles. That can also lead to a couple of mistakes.
Subcontractors are also less likely to point out all the issues in the code they delivered. In particular when it implies additional work that may not be paid. In other words, involving a third party to test the performance can be a good way to challenge the development team.
Does Relevant load testing relies on a proper methodology?
It already takes a lot of time for a pure tester to master it. On average, a tester should need between 1 and 6 month to get proper methodology. That’s because it takes some trial and error to find the good mix between realism and preparation time. And that’s just to cover the basics of load testing. We all learn new things for our entire careers. Anyway, it seems unrealistic to expect a developer to have the time to do the same. Especially since he has other tasks to perform. As a consequence, tests are often rushed. And they results are of course questionnable.
Devops already assume a lot of roles.
Adding another skill to the list will not help. It only adds more pressure. A devops is already a jack of all trades, don’t make him a master of none.
Finding the Unicorn
Good devops profiles are hard to come by.
And looking for one that also masters the testing process only makes it worse. Even if you manage to find one, you won’t be able to replace him when he leaves. Or at least it will take time. Time that is not spent preparing a smooth transition.
Load testing an application requires a larger skillset than any individual can master. But the most important skills are the one that allow you to properly test the application. Which already involves a mix of methodology and technical skills. For the rest you shoud rely on other people. The key is to get them involved in the early stages of the test campaign.
He is the decider, he will help define objectives and validate deliverables. The manager shouldn’t be involved too much in the testing process. But whoever plays this role will have to be able to understand the results. This is something we will discuss in the reporting section.
Most importantly his role is key to avoid deadlocks in the testing process. He can help with long internal processes or synchronization with other teams.
It is easier to define test scenarios when you have some functional knowledge. When you don’t, working with a functional expert will make the test scenario creation faster.
Still, keep in mind you don’t want to test all the functional use cases. Because functional experts are used to exhaustive testing of all functionalities. We will discuss this further in a later section of this training. But for now just keep in mind that load testing scenarios are more expensive to develop. So we must focus on the most relevant features.
Whether or not you need the help of a developer highly depends on:
- The complexity of the application,
- The level of documentation available,
- And your technical skills.
In any case a developer can help when designing the test scripts. Since he knows how the application communicates.
Here comes infrastructure technical knowledge. And as it is very unlikely one person can master all these technologies, we are probably referring to a team of experts.
You want him or them involved as soon as possible. In particular to discuss what to monitor and how. They could also have some insight on the objectives since they might suspect potential issues before the tests.
And as a good practice, have them check the infrastructure configuration prior to the tests. Most often than not, servers and applications use the default configuration. Which means the resources of the servers might not be available to your JVM, database or web server.
To sum up all these profiles can help greatly during a test campaign. You could even be one of them. And you would rely on the others to get the job done.
Test scenarios define what activities your tests will perform. A single test scenario is usually a list of actions in the application’s interface. It has to mimic the behavior of a real user. Before going further we will focus on the test scenario selection process.
In functional testing, every possible situation must be controlled. You end up having many scenarios to cover all these possibilities. In load testing we only care about generating the right load to the servers. And you can achieve that through much less use cases.
Less use cases means less work, time and budget.
Or more time to analyze and optimize the application. But to achieve that you must select the proper uses cases. Otherwise your results could be compromised because you did not test a critical feature. In the end, the more accurate your selection, the more accurate the tests results.
For instance, adding a XL T-shirt to your cart or a XS one does not make any difference in terms of performance. Visiting product page A or product page B neither. You can randomize these actions to avoid cache effects but they do not require another use case. A login on the other hand has more consequences. You might collect more data on logged in users than guests.
The shipping mode could also have an impact since different third party sites will participate. This is the kind of actions you want to identify and include in your performance test scenarios.
A good rule of thumb is to keep the number of test scenarios between 1 and 5 whenever possible.
A popular saying is that 20% of the use cases will generate 80% of the load. This principle does not apply only to performance testing. It is called the Pareto principle. It states that 80% of the effect can be obtained through roughly 20% of the causes. And the key takeaway in our situation is to keep the load testing scenarios compact. By compact, I mean:
- Horizontally: limit the number of test scenarios,
- Vertically: limit the complexity of each test scenario.
We will talk more about the reasons behind this choice later. For now let’s just say that compared to functional tests our test scripts have less resources to run. Since we want to run many of them concurrently, we have to pay more attention to their optimization. But as a consequence, they are more complex to create than functional scripts.
Also the timeframe associated with performance tests is usually short. Because of this having too many test scripts can stall you completely. The longer you take to develop them, the more likely you are to face a change. And change in the application can force you to start over from scratch. Which gives more time for the application to change, etc…
Trust me, you don’t want to be trapped in such an downward spiral.
For this quick exercise find what’s wrong with each test scenario: