Selecting the test environment is the second challenge of most test campaigns. It can be easy in some situations. For instance when there is only one environment. Because not everyone can afford a preproduction environment. Or when the production is a brand new environment dedicated to the tests. In this case, you can skip this topic because future production will be the best environment. But in other situations, which environment is more relevant?
First, the tests will be different depending on the environment. That is because it may give you access to more realistic data. Or the different processes may be closer to reality. And of course this will have a huge impact on the preparations.
In a development environment the application may not have its final form. It’s therefore difficult to conduct complete tests. In particular since there might be no interface at this point. Instead, a good strategy is to test components one by one. That means testing the backend services directly. Or direct calls to the database.
The purpose is to get performance metrics as early as possible. First looking at the services response time gives an idea of the final response time. It may not be very accurate yet. But at this stage, an idea is better than nothing. In case a service takes 1 second to respond you can measure its impact on the finished application. Just find each page using it and see if their response time would still be acceptable. This way you can already decide if have to work on optimizing it.
These tests can then be conducted on every new build to see if performance checks out. If the results are very different, investigations can start right away. That is before the delivery in later environments makes it harder to fix. This should make fixing performance defects easier.
Now of course all this process makes sense on large projects. You might not want to invest so much on a simpler application.
But keep in mind that defects you detect later will cost much more to fix.
Preproduction environments are more likely to host a complete version of the application. The main drawback is that it will not be available until later development stages.
But keep in mind that component testing does not replace a complete test. You must test the application when all services come together. This is the only way to uncover all performance defects.
That being said, what is so special with preproduction?
First, preparing test data will be much easier. You should be able to easily create any information you need in the database. Keep in mind that before you do so, it is important for the database to contain realistic amounts of data. Otherwise you could miss some issues. In particular regarding indexes.
Load tests have an impact on everyone using the application. This makes it easier to schedule a load test in pre-production. You should be able to have the application all to yourself during business hours. It means the cost of scheduling tests will be much lower.
Test 3rd party providers
Let’s take an example. You are testing an e-shop that relies on a third party to handle the payment. In preproduction you are likely to have a test version of the payment in place. This way you can actually run a test script that will proceed with the payment. That would not have been possible in production since the payment would have to be real. Which is why a test in preproduction will be more realistic when third parties are involved.
Production environments usually share some hardware with each others. It can be the network link with the outside world, the firewall, or the physical machine behind your Virtual Machines. Because of this tests can fail from external causes. Even worse, when testing outside of business hours, you might not see the failure at all. In pre-production, you might have a lot of mutualization, but testing during the day can point out real issues.
To prepare your test scripts and run tests, you need a stable environment. Which means no new version until your tests are done. This is easier to achieve in production since updates should not occur on a regular basis. Or at least less often than earlier environments.
Infrastructure and security teams must be aware of what you plan. Otherwise they might react to your tests as an attack. They could invest a lot of resources to find out what happens and fix it. Instead, having them on board will ensure they can deactivate whatever protection there is. But that will also make scheduling the tests more difficult. Since these protections can only be deactivated temporarily.
For obvious reasons, a proper load test will be difficult to launch during the day You don’t want to slow down users, risk a shortage, etc… Production tests are most of the time run at night. Scheduling one will take more time since everything needs to be prepared. And this includes server monitoring, data to be used by each test, recovery procedure, etc… Otherwise you would have to involve a lot of people and thus increase the cost.
Some steps of your tests scenarios might not even be possible to run. We’ve discussed the case of payment, but it goes beyond that. For instance Can you subscribe thousands of fake customers in production? That might create issues with other teams or third parties. You will have to at least clean up after the tests. Also still looking at an e-shop what about the products you add to the cart? More specifically when dealing with flash sales. You might end up putting all products in carts of fake users. And real users would not see them until they expire. Most sales happen in the first minute of a flash sale. SO this is a negative impact you want to consider.
In the end, production will always be the most realistic environment. And here I mean in terms of hardware and server configuration. Pre-production wight be close. But there’s no telling what will happen after you double the amount of servers. If that’s the difference with production. And even worse, it might not be so simple. Differences are not only in the number of servers, but also their configuration. Both for hardware and software.
Find out which tests can be launched on which environment in this configuration: