Should I test my external providers
If you ever recorded a load testing scenario on a website you probably noticed that the list of external links can be scary:
Plus these links might occur several times, making your user profile difficult to read. Because of this even a simple task like separating the different steps into distinct transactions can become tedious:
Every time I work on such tests the question is always the same: “Do we need to simulate these requests?"
Why so many?
I once did a test campaign on an application which was referring to over a hundred of different hostnames from its homepage. When you look at the list, most of this external hosts have a purpose:
- Statistics websites
- Cache & CDN
- Integration (Facebook, twitter, etc…)
- External ads
- Etc…
The problem in this particular case was that there were several of each, which does not makes much sense. But even if you try to optimize the number of third party providers, you will probably end up with about 20 different hosts.
But I just want to test my application!
You might think that filtering everything apart from requests to your own website is the best way to go. Well yes, but users coming to the website will load all these external links. To get a realistic idea of the user response time, it is then important to simulate them. Also you have little to no idea of how well they handle the load.
They might be badly configured thus slowing the page load. So ideally we want to test them all, but there are some practical reasons why we can’t always do this.
Complexity
I already mentioned it but a virtual user profile containing dozens of external links can become difficult to read. This of course translates into more time to get it functional, while being more error prone. On top of that these external calls usually contain a lot of dynamic parameters to correlate if you want to get them working.
I would say that on average, on a very simple website a test case can be done in an hour. These external calls can take up to another few hours each to configure. In that case is it still worth keeping them all?
Other reasons
You might also want to consider the following:
- Some of these external links might point to paying services that will cost you more because of the load tests. In particular if you pay for every hit.
- The statistics websites will consider the load tests as part of the real traffic on your website. Because of this you will get wrong usage statistics that can lead to wrong decisions later.
- Requests to a CDN depend on your geographical location. For example, if you record a virtual user profile in one country and replay it in the cloud from another you should not leave the recorded calls because they might point to the wrong country.
- Some external providers will blacklist you if you generate too much load. From an external point of view, a load test can be seen as a DDOS attack.
Testing the test
In most situations to decide whether you should or shouldn’t include some external providers, it is best to run a quick test. First consider the ‘safest’ providers, the ones that have a very quick response time or that you trust the most. For instance when testing 1000 VUs, the calls you make for facebook integration are not likely to overload them.
Then, some of these requests might be running only after the page loads. That way they are not hampering the overall performance unless you wait for absolutely every request of each page to load. Google analytics recommends to do this for instance. The application developers can help you greatly to figure these out.
Conclusion
In the end you should balance complexity and relevance to decide whether you must include some external links in your tests or not. Some will be easy to rule out because they have no impact on performance or you won’t have any impact on their performance. For the others a quick test might tell you how relevant they are, but in the end you will have to decide how much effort you are willing to put in your testings.
