When we consider performance testing most of the focus of the approach is placed on the script creation activity and making sure that requirements are covered, these are very important parts of your performance testing approach but so is the Test Execution.

The frequency at which you run your tests is important and can save you time and effort in your script maintenance.

A good performance testing execution strategy gives you the maximum amount of benefit it can by finding performance issues early.

We really believe that test execution should be a daily activity and not something that should not be done:

• At the end of a development cycle,
• Before go-live,
• When it is scheduled on a project plan,
• When it feels appropriate.

Daily execution is not an overhead and becomes a cost and time saving approach if managed correctly. Before we look at the subject in further detail we wanted to show you a diagram taken from an article posted by James Bach who is part of a community called the Context-Driven School and has created a methodology called Rapid Software Testing.

Where quality comes from Inexpensive low level testing and this should include performance testing.

The article can be found here Round Earth Test Strategy and is worth a read.

Script Creation and Maintenance

What has script creation and maintenance got to do with test execution you may be thinking?

The answer is that you have to recognise that script creation and maintenance is something you are going to have to do on a regular, if not daily, basis and therefore these activities are tightly coupled with the execution activity.

Your application will evolve and change as you are effectively executing performance tests during the application development iterations and therefore new application functionality or endpoints will be regularly delivered into QA and require new tests and existing application code or endpoints will be evolving and require maintenance to existing tests.

• If this is done daily the overhead is kept to a minimum.
• If you leave significant periods of time between test execution cycles the likelihood that your tests will require extensive maintenance or even re-writing increases.

There is little point in investing a significant amount of time and effort in writing well thought out performance tests if when you come to execute them they fail, regular execution will remove this risk.

Out of hours execution

So we have said that daily execution should be your normal approach to performance testing and running out of common working hours provides a number of benefits.

You are not spending time in the day executing tests and can use your day to work on other aspects of performance testing, we will discuss what these other activities may be later.

You are executing your tests against a stable environment and it is likely, depending on your organisations ways of working that changes are being made to components in your test environments regularly throughout the course of the day as developers push code.

Out of hours execution minimises this disruption.

Executing out of hours will maximise the chance that each set of tests will run against a consistent load on the system. During the working day there may be other aspects of QA all sharing the environment making the load profile inconsistent.

Your daily routine then may become:

• Analyse results,
• Feedback to development teams,
• Fix and tests that may have failed because of data or changes to functionality or services,
• Build any new tests and add to your overnight execution schedule.

A final benefit of your out of hours execution schedule is that once you have analysed your results if you determine there has been no regression and there have been no failures and no new functionality needs to be added then you can spend the remainder of your day doing things that provide even greater performance benefits.

The reality of daily execution is that on many occasions there will be very little to do in terms of script creation or maintenance and the opportunities to add value outside of just execution are regularly available.

We will discuss these benefits later.

How do we execute

So we have said that out of core hours execution is the best strategy but how do we do this?

There are a number of options and it does to a degree depend on your performance tool of choice and your operating system, these include:

• Jenkins pipelines,
• CRON Job,
• Some commercial tools include scheduling functionality,
• Windows Scheduler.

The method you use does not really matter, it is the fact that you are using this dead-time to execute tests will provide you with much more time to address other aspects of performance testing especially when it comes to the longer running Soak tests.

Benefits

There are some obvious benefits of regular and constant execution of performance test and these include:

• A constant awareness of how your application is performing.
• You can build your tests as your technology stack evolves.
• Make performance improvement recommendations during the development cycles when it is easy to fix, or improve.
• Help define hardware requirements for your platform as the testing evolves.
• If performance regresses you can pinpoint to a release, or a code check-in, or a particular component that caused the issue vastly improving and speeding up the root cause analysis.

There are however some not so obvious benefits and these should not be overlooked, these include:

More time to analyse results

Regular execution means that you are not spending a significant amount of time looking at performance but just a small amount each day.

You may find some days there is nothing beyond a check that all tests have completed to do, your analysis becomes a very easy activity as you are doing it daily.

The difficulty with periodic execution is that normally the performance testing window falls just before go-live and you run the risk of being unable to spend as much time analysing results as you would like meaning that you may miss something that will cause performance issues in production.

Better quality performance framework

A framework that supports your company’s performance testing strategy is important but often you have very little time to adapt or update your framework as your test cycles are relatively short and, as already discussed, at the end of the delivery lifecycle when timescales are tight.

Daily execution allows you to make changes to your execution framework on a regular basis; this refinement only goes to make your performance testing even more efficient.

Spot regression and trends

Having daily results allows you to easily check for trends in response times over a short period of time and not rely on comparing results from a test executed a long time in the past.

Having that awareness of the performance of your application daily makes it easy to spot irregularities in your results.

More time to analyse CPU, Memory and DB Statistics

While response times and concurrency levels are more than likely the measure you are interested in to meet your requirements it is always good practice to determine how much impact your high levels of load have on your infrastructure. Regular execution gives you more opportunity to analyse CPU and Memory and see whether there is any opportunity to improve SQL or suggest database table indexes.

If you do increase, or decrease, the amount of CPU or Memory, or change your Garbage Collection policy then the results are available next day for you to compare against. Any recommendations for changes to infrastructure can be fed back to the infrastructure teams to ensure your production environment is built in line with your recommendations.

Resilience tests

All non-functional requirements need to include a measure of resilience whether this is high availability or active passive or running in a cloud based solution where containers can self-monitor and re-start on failure.

Our daily execution strategy allows you time to do this and experiment with various failure and recovery scenarios with results analysis against the previous days results easy to do and easy to spot the impact of the failure.

More opportunity for value add activity

Performance tests do not have to always be used for performance testing, this may seem an odd thing to say but, if built in a framework that abstracts the data driving the tests into flat files they can be used for many other purposes.

A data abstraction example for JMeter exists here Flexible Test Plans although other tools will follow a similar pattern.

Examples of other QA related activities that these tests can be used for include:

Data Creation

Adding test data to test environments to either support performance testing, functional testing or business user testing.

• If you need data on a regular basis in a regular format then automation is the way to create this.
• If you have a performance test that does this then use it as the daily execution schedule means the test is always maintained in line with the current code base.

Smoke Tests

Checking that a deployment has been a success is another job for automation, a smoke test is normally executed immediately after the deployment process.

Your performance tests can be run as a single user executing a single iteration to achieve this, if you have a performance test then use it.

In both the above examples the critical thing is if you have an existing test then this can be used, writing tests that do not provide benefit for performance testing should be avoided as these will need to be maintained and this will be an overhead you do not need.

We are not agile!

We are conscious that many organisations still use a waterfall approach for technology delivery and development and just because you may not be a fully agile organisation does not mean you should not adopt agile performance testing.

Formal test cycles

While you are running tests on a daily basis there still may be a need for a formal performance testing phase.

It may be important to the programme and key stakeholders that a set of tests are run on the final build before promoting the code to production and this may be in the form of:

• Peak Volume,
• Soak Test,
• Scalability Test,
• Resilience Test,
• etc.

So what is the point of all this early test execution when you still need to run a formal test cycle to pass your programme quality gate?

• Regular execution of your test will provide a high degree of confidence that these formal tests will pass with no issues.
• There will be no script maintenance needed as you will have been constantly maintaining your tests through regular execution.

Conclusion

There are no negatives to daily execution only positives and to move away from a culture where performance testing is rushed and provides an important but limited benefit can only improve your end product.

You have to build performance tests and you have to execute them so why not do this incrementally instead of rushing to build a performance test that is executed periodically and therefore becomes quickly out-dated and not fit for purpose.

The benefits are numerous and also provide value outside of pure performance testing if you try and leverage the tests for other purposes as we have discussed.

To paraphrase the diagram we started with:

“Quality above requires quality below and inexpensive low level testing reduces dependence on expensive high level testing.”

Performance testing is inexpensive low level testing, and the more often you execute it the more benefit it provides.