Should I test my external providers
If you ever recorded a load testing scenario on a website you probably noticed that the list of external links can be scary:
If you ever recorded a load testing scenario on a website you probably noticed that the list of external links can be scary:
Plus these links might occur several times, making your user profile difficult to read. Because of this even a simple task like separating the different steps into distinct transactions can become tedious:
Every time I work on such tests the question is always the same: “Do we need to simulate these requests?"
I once did a test campaign on an application which was referring to over a hundred of different hostnames from its homepage. When you look at the list, most of this external hosts have a purpose:
The problem in this particular case was that there were several of each, which does not makes much sense. But even if you try to optimize the number of third party providers, you will probably end up with about 20 different hosts.
You might think that filtering everything apart from requests to your own website is the best way to go. Well yes, but users coming to the website will load all these external links. To get a realistic idea of the user response time, it is then important to simulate them. Also you have little to no idea of how well they handle the load.
They might be badly configured thus slowing the page load. So ideally we want to test them all, but there are some practical reasons why we can’t always do this.
I already mentioned it but a virtual user profile containing dozens of external links can become difficult to read. This of course translates into more time to get it functional, while being more error prone. On top of that these external calls usually contain a lot of dynamic parameters to correlate if you want to get them working.
I would say that on average, on a very simple website a test case can be done in an hour. These external calls can take up to another few hours each to configure. In that case is it still worth keeping them all?
You might also want to consider the following:
In most situations to decide whether you should or shouldn’t include some external providers, it is best to run a quick test. First consider the ‘safest’ providers, the ones that have a very quick response time or that you trust the most. For instance when testing 1000 VUs, the calls you make for facebook integration are not likely to overload them.
Then, some of these requests might be running only after the page loads. That way they are not hampering the overall performance unless you wait for absolutely every request of each page to load. Google analytics recommends to do this for instance. The application developers can help you greatly to figure these out.
In the end you should balance complexity and relevance to decide whether you must include some external links in your tests or not. Some will be easy to rule out because they have no impact on performance or you won’t have any impact on their performance. For the others a quick test might tell you how relevant they are, but in the end you will have to decide how much effort you are willing to put in your testings.