About 8 months back, after a few years working on the same technology stack used for OctoPerf, I wanted to check if we should follow the latest trends: micro-services and reactive programming.

Re-building OctoPerf with other technologies, just for a POC, seemed way too long. After all, the development of the frontend alone took me a few years, not to mention the backend … OctoPerf comes with really advanced features regarding load-testing!

So, I needed to create an application that would be complex enough to check the suitability of a new technology stack for OctoPerf, while also not taking more than a few months to develop as a side project: Welcome to Kraken!

Keep reading to know how I started by creating a Docker management application to end up creating a load testing IDE.

Kraken Administration

So my first idea for a proof of concept is a generic Web Application capable of managing Docker containers. Then, depending on what container is started, this application can run anything that is packaged in one or several Docker images.

This Kraken Administration app can:

• Start, stop, and list Docker containers,
• Manage Docker images (List, pull and remove),
• Start shell scripts and run docker-compose commands,
• Manage system configuration files such as .sh or docker-compose.yml.

All these features are available through a nice UI made to look like an IDE.

Kraken’s UI is inspired by an application that I use daily: IntelliJ Idea, hence the Dark theme.

Like any other IDE it comes with:

• A central panel dedicated to code edition,
• Side panels that can be collapsed and resized,
• A file tree used to browse the code hierarchy,
• etc.

Microservices

The first thing I wanted to experiment is micro-services:

Microservices is a software architecture style from which a complex set of applications is broken down into several independent and loosely coupled processes, often specialized in a single task. Independent processes communicate with each other using APIs independent of the programming language.

Docker really helps with the deployment of microservices. You can package each process in a Docker Image.

For Kraken, several images are used:

• octoperf/kraken-executor: a Java backend to run docker and shell commands on the host machine (via a REST API),
• octoperf/kraken-storage: a Java backend that exposes the file system through a REST API and allows for files manipulation,
• octoperf/kraken-administration-ui: an Angular frontend application that provides an IDE-like UI that calls both backend APIs,
• haproxy: used to serve all REST APIs and the frontend on the same port (80).

Now that I think of it, it might be a good idea to split the Executor backend into several containers : one to run Docker commands, one to run docker-compose commands, and a last one to run shell commands.

Gradle

Starting over with a new Java application is also for me the occasion to test a new build tool. I have quite some experience with Maven. Build times being too long for my taste I decided to give a try to Gradle.

Gradle is also more flexible, but I must admit I struggled to create a multi-project build with it. Apart from that, it’s really easy to understand and use. Many plugins are also available, but the most important benefit for me is its build time performance. I come back in details on Gradle in another blog post: Gradle multi-project builds.

DooD

Kraken not only runs on Docker, but it is also capable of running other Docker containers on its host machine. This can be done using either Docker In Docker or Docker Outside Of Docker.

I ended up using DooD, after a hard fight with volumes permissions. This topic deserves a blog post on its own, and so it will be.

Finally, I may check Kubernetes and Docker Swarm in the near future, to start multiple Docker containers on remote hosts.

Going Reactive

Kraken follows the reactive programming paradigm from the deepest Java service to the highest frontend component.

Reactive programming is a programming paradigm designed to maintain overall coherence by propagating the modifications of a reactive source (modification of a variable, user input, etc.) to the elements dependent on this source.

To do this, it uses the following stack:

• Java 11,
• Spring Reactor,
• Spring WebFlux,
• Server-Sent Events,
• Angular.

Angular

Having the experience of creating OctoPerf AngularJS frontend, the simplest part for me was certainly to switch to Angular. We already use TypeScript for OctoPerf, so the main difference is that Angular relies on RxJS from server-side communication with the Http client to the reactive forms.

Compared to AngularJS, it is a big improvement in terms of usability and maintainability.

That being said, I don’t feel like switching to Angular for OctoPerf yet. Some features like tree views are not as complete. Also, we use Bootstrap, and I’m not sure how we could make it get along with Angular Material.

Server-Sent Events

The most challenging part here is to make Spring WebFlux return data according to the SSE specifications and call it from Angular. But once working, it allows propagating modifications made on the file system or command execution logs up to the UI. I’ll get back to that in another post blog.

Spring WebFlux does not work with Spring MVC. So, using it for OctoPerf might prove difficult as we would have to split our backend in micro-services to make the changeover gradually, or completely replace Spring MVC by WebFlux in one go.

One last thing regarding Server-Sent Events: It is not supported by Internet Explorer / Microsoft Edge. What a bummer! I will have to use polyfills.

Edit: Please check my blog post on reactive SSE for more information.

Kraken Gatling

To ensure that micro-services really give more flexibility to my POC, I wanted to add some features specific to load testing and check if that could be done without too much modifications to the existing Docker images.

Having already heard about the Gatling/InfluxDB/Grafana stack, I thought it might a good idea to try to append it to Kraken.

This led to the Kraken Gatling load testing application:

HTTP Archive import into Kraken Gatling

This application is used to facilitate load and performance tests made with Gatling, from script recording to metrics analysis.

In Kraken Gatling you can import HAR files, convert them into Gatling .scala test scripts, and compare the recorded requests/responses to debug ones. And of course you can also quickly run load tests and visualize results (during the load test execution) thanks to Grafana.

Y’all Got Any More Of Them Docker Images

Technically, I had to add several Docker images to the application stack:

• First I started by creating a second frontend (I had to split the existing project into libraries),
• Then I created a kraken-analysis Java backend that wraps calls to the two other Java backends and to the analysis stack,
• I also configured a NGinx server kraken-static to serve Gatling’s report and debug responses as HTML content,
• Finally I added the load testing analysis stack:
  • kraken-influxdb to store the metrics,
  • kraken-grafana to display performance dashboards,
  • kraken-analysis-telegraf to push monitoring metrics about the Docker host machine.

I also created several other Docker images that are only started when needed: kraken-gatling-recorder to record HAR, kraken-gatling-debugger to debug and kraken-gatling-runner to execute load tests.

Automate everything

As the number of micro-services grows, the development process becomes a bit annoying if you need to manually test every modification:

While everything is packaged in Docker images and started automatically thanks to a docker-compose.yml script in production, you have to start each required service by hand before doing your test using a dev environment.

For example, to test the Kraken Gatling frontend, you need to start the Executor backend, the Storage backend, the Analysis backend, the Static backend, InfluxDB, Grafana, etc.

To ease this process, we have the habit to use makefiles to automate everything. So running an application is usually done with one single command like make serve-storage.

Unit testing

Also, we heavily rely on Unit Tests to have good code quality. This also ensures that any modification made to the backend applications works as expected without having to start the frontend and test it by hand. Even though I did not achieve the same level of code coverage on Kraken than what we did on OctoPerf, it’s still pretty good:

• Jacoco and SpotBugs are used for Java applications,
• Karma and TSLint are used for Angular applications.

Open Source

Kraken’s code is Open Source and available at https://github.com/OctoPerf/kraken.

If you want to know more about the Docker management application, please check out the Kraken Administration documentation page.

To learn more about a simplified Gatling load testing experience, please read the Kraken Gatling documentation page.

Conclusion

What started as a proof of concept is now an application. I did not intend to create Kraken while starting to test new technologies, but one thing leading to another, here it is! I will surely continue to develop Kraken as a side-project to OctoPerf.

The good thing with Kraken is that is can be used to run any tool provided:

• It uses the file system to store its data,
• It can be run inside one or several Docker container(s).

It’s even easier if it pushes metrics to InfluxDB, you can quickly get a nice Dashboard in Grafana.

Some other technical blog posts will be coming soon:

• A guide on Gradle multi-project builds for Maven users,
• A tutorial to create Angular libraries and a multi-applications project,
• An article on using Haproxy to serve several Angular fronts as well as the required backends,
• A post on Sever-Sent Event, Spring WebFlux and Angular,
• A devops tutorial to automate build processes with makefiles,
• UX/UI guide to make an IDE-like web application,
• etc.

Your feedback is welcome, please leave a comment on this blog post or open a issue.