JMeter VS Gatling Tool
By Jérôme Loisel - CTO, on .

Categories: Tags: gatling jmeter

JMeter VS Gatling Tool

Following up our article about why we chose JMeter to build OctoPerf, our Cloud Load Testing Platform, let’s now compare Apache JMeter to Gatling Tool in details. Both tools are great, but they excel in different fields. We aim to give you the best possible overview of these tool and try to stay as objective as possible. The purpose of this post is to give a technical insight about how each of these load testing tools are made.

History

This section covers historical data about these two open-source projects.

JMeter

JMeter has been first released in 1998, so this project more than 15 years old. It’s a mature project which has released more than 20 versions. JMeter is purely written in Java, although OpenHub shows that it also contains XML and HTML:

JMeter Languages

JMeter has more than 10.000 commits representing almost 230K lines of code. 33 people contributed to the project so far. Philippe M. is one of the biggest contributor to the project, mostly in Java. He is the founder of Ubik Load Pack, a French company offering commercial JMeter plugins.

Top 3 JMeter Contributors

JMeter is open-source and released under the Apache 2.0 license.

Gatling Tool

Gatling Tool has been first released on December 20, 2011. Gatling is a younger project, with almost 4 years old codebase. Gatling is mostly written in Scala.

Gatling Languages

Scala differs for Java by offering full support for Functional programming. Scala has been created by a Martin Odersky, as German computer scientist.

Stéphane Landelle is the main contributor and creator of Gatling tool. More than 60 people contributed to this project. Its size is almost 31K lines of code.

Top 3 Gatling Tool Contributors

Gatling tool is open-source and released under the Apache 2.0 license. Its development is supported by Excilys.

History Comparison

JMeter is older than Gatling tool with a well-established code base. Gatling tool has less lines of code than JMeter, but it can be explained by the fact Gatling is written in Scala. Scala is way more concise than Java.

Documentation

The match between JMeter and Gatling tool ends with a draw on documentation. Both are great! You may prefer the look of Gatling’s documentation, but JMeter has recently made a great effort to refresh the look of their website.

JMeter

The JMeter documentation is really great. Every feature of the tool is detailed with every possible setting. They truly made a huge effort on explaining how JMeter works, how to create your first test and more.

Gatling

The Gatling Documentation is very clear and concise, like the philosophy of Scala. There is really nothing more to say, the documentation talks about any feature in depth and covers the whole tool.

Installation

JMeter and Gatling tool both require a Java JRE installed on the machine.

Both tools can be installed by doing the following steps:

Both are heavy clients that must be installed as a standalone program on your machine. There is no real difference in installing JMeter or Gatling. Both are shipped as archives, no additional installation is required once you have a JRE.

Project syntax

One big difference between JMeter and Gatling is the project format. JMeter saves projects as an XML file with jmx extension, whereas Gatling saves them (called Simulation) as Scala classes with scala extension.

Here is what a JMeter JMX looks like:

{% gist jloisel/4cc64b7f71ab2957367e %}

Here is what a Gatling simulation looks like:

{% gist jloisel/1cd405b2618b15ae4b73 %}

JMeter JMX is more verbose than Gatling Scala simulation. But, you’re not intended to read the JMX file directly. JMeter has a GUI to open those file, and display them. Gatling simulations are expected to be edited and modified directly.

GUI VS Code

The real battle between JMeter and Gatling tool starts here. One tool has a complete GUI to edit and run scenarios, whereas the other online runs as command-line tool.

JMeter: GUI

JMeter is GUI oriented. Opening, editing and saving a JMX file must be made with JMeter Graphical interface. Do not attempt to modify the JMX yourself, even if the XML is human readable, it’s really not sustainable. JMeter’s GUI probably suits well for people accustomed to graphical tools.

Here is what JMeter graphical interface looks like:

JMeter GUI

Gatling Tool: Command-line

In contrast to JMeter, Gatling Tool has no GUI. Editing the simulation requires you to write code. Gatling tool is clearly made for developers, although they claim it can be used by non programmers too. The Gatling simulation Scala syntax is pretty easy to understand, but as a developer so point of view is a little biased.

Running a Gatling simulation:

{% highlight bash %} ./bin/gatling.sh -s basic.ExampleSimulation {% endhighlight %}

Performance: Synchronous VS Asynchronous

JMeter uses the One Thread = One User Paradigm. It means that each simulated virtual user inside the load generator equals to a single thread. Wait times are simulated by Thread.sleep. The user waits for the server response when sending a request. This is sometimes called Synchronous processing.

Gatling uses a more advanced engine based on Akka. Akka is a distributed framework based on the actor model. It allows fully asynchronous computing. Actors are small entities communicating with other actors through messaging. It can simulate multiple virtual users with a single Thread. Gatling also makes use of Async HTTP Client.

There was a debate about JMeter versus Gatling performance. As of today, none is performing better than the other. No test has shown a clear advantage of JMeter over Gatling or the inverse.

Single VS Distributed

When load testing an HTTP application, once you need to simulate several thousands users a single machine may not enough. The load must be splitted among multiple machines. JMeter and Gatling tool are clearly not behaving the same way on this field.

JMeter is distributed. JMeter uses a master / slave architecture. The master is connected to slaves via the RMI protocol. The master sends the JMX to simulate to each slave. Each slave plays the JMX. If you have 6 slaves, and your JMX simulates 1000 users, then you end up playing 6x1000 = 6000 users. Results are sent back to the master for consolidation.

If you want more info, see the Distributed Testing with JMeter tutorial.

Gatling is single host. Although Gatling is based on Akka which is a distributed actor framework, Gatling is not distributed. It cannot scale horizontally. You can of course launch multiple Gatling instances on different machines at the same time, but it’s fully manual. You will also need to consolidate the results from the multiple injectors manually.

Distributed load testing is planned for Gatling, but only as a commercial feature.

Recording a scenario

JMeter

JMeter has a built in proxy recorder to generate the requests from the user navigation.

JMeter Proxy Recorder

Gatling

Gatling has a separate Web proxy recorder, which works the same way as the JMeter recorder. It generates a Scala simulation from the recorded user session. It has an interesting feature which allows to convert an HAR archive into a Gatling simulation.

Gatling Proxy Recorder

Running a test

JMeter

To launch a test with JMeter, there are two methods:

  • Launch the GUI, then launch the test inside GUI,
  • Launch the test via command-line (recommended).

JMeter offers various ways to view the results when using the GUI, for example with a graph:

JMeter - Test Result

But the recommended way is to launch JMeter in command-line, and save results as JTL files. Using GUI listeners consumes a lot of memory.

The best way is to use a BackendListener which exports results to Grafana or InfluxDB:

JMeter - Backend Listener

Gatling

To launch a test with gatling, run the .sh script and select the simulation (within user-files/simulations folder):

Gatling - Running a test

During the test run, gatling displays statistics within the console. The results of the test can be viewed at the end of the test. Gatling generates a generic test report with response time, number of OK/KO transactions and detailed statistics for every request. The report is not customizable.

Gatling - Test Result

Cloud Load Testing

There are multiple companies that provide Cloud Load Testing for one or the other tool. All of them are just providing a facility to scale tests into the Cloud and provide a basic results reporting. You upload your JMX scenario or Gatling simulation to those providers, and they run your scenario on Cloud machines.

We are the world first JMeter Saas Company, providing 100% Web JMeter Scenario Design. We also Scale from multiple locations and provide fully customizable test reports.

Protocols

JMeter simulates many protocols like HTTP, JDBC, LDAP, SOAP, JMS, and FTP. Gatling Tool supports HTTP, JMS, Websocket and SSE (Server Side Events) at the moment. (version 2.1.x)

Which Tool

It’s not really a match between JMeter and Gatling. Each of them aim different audience.

Choosing JMeter or Gatling tool really depends on what you are comfortable with. If you’re a developer who likes to code, you will probably use Gatling. If you are a performance tester, more interested into GUI tools, JMeter seems to suit you well.

Both tools are great, although they require some learning to get them up and running smoothly.

Related Content