About 8 months back, after a few years working on the same technology stack used for OctoPerf,
I wanted to check if we should follow the latest trends: micro-services and reactive programming.
Re-building OctoPerf with other technologies, just for a POC, seemed way too long.
After all, the development of the frontend alone took me a few years, not to mention the backend …
OctoPerf comes with really advanced features regarding load-testing!
So, I needed to create an application that would be complex enough to check the suitability of a new technology stack for OctoPerf, while also not taking more than a few months to develop as a side project: Welcome to Kraken!
Keep reading to know how I started by creating a Docker management application to end up creating a load testing IDE.
Kraken Administration
So my first idea for a proof of concept is a generic Web Application capable of managing Docker containers. Then, depending on what container is started, this application can run anything that is packaged in one or several Docker images.
This Kraken Administration app can:
- Start, stop, and list Docker containers,
- Manage Docker images (List, pull and remove),
- Start shell scripts and run docker-compose commands,
- Manage system configuration files such as
.sh
or docker-compose.yml
.
All these features are available through a nice UI made to look like an IDE.
Kraken Gatling Application Overview
Kraken’s UI is inspired by an application that I use daily: IntelliJ Idea, hence the Dark theme.
Like any other IDE it comes with:
- A central panel dedicated to code edition,
- Side panels that can be collapsed and resized,
- A file tree used to browse the code hierarchy,
- etc.
Microservices
The first thing I wanted to experiment is micro-services:
Microservices is a software architecture style from which a complex set of applications is broken down into several independent and loosely coupled processes, often specialized in a single task.
Independent processes communicate with each other using APIs independent of the programming language.
Docker really helps with the deployment of microservices.
You can package each process in a Docker Image.
Docker
For Kraken, several images are used:
- octoperf/kraken-executor: a Java backend to run docker and shell commands on the host machine (via a REST API),
- octoperf/kraken-storage: a Java backend that exposes the file system through a REST API and allows for files manipulation,
- octoperf/kraken-administration-ui: an Angular frontend application that provides an IDE-like UI that calls both backend APIs,
- haproxy: used to serve all REST APIs and the frontend on the same port (80).
Kraken Admin Architecture
Gradle
Starting over with a new Java application is also for me the occasion to test a new build tool.
I have quite some experience with Maven.
Build times being too long for my taste I decided to give a try to Gradle.
Gradle is also more flexible, but I must admit I struggled to create a multi-project build with it.
Apart from that, it’s really easy to understand and use. Many plugins are also available, but the most important benefit for me is its build time performance.
I come back in details on Gradle in another blog post: Gradle multi-project builds.
DooD
Kraken not only runs on Docker, but it is also capable of running other Docker containers on its host machine.
This can be done using either Docker In Docker or Docker Outside Of Docker.
I ended up using DooD, after a hard fight with volumes permissions.
This topic deserves a blog post on its own, and so it will be.
Finally, I may check Kubernetes and Docker Swarm in the near future, to start multiple Docker containers on remote hosts.
Going Reactive
Kraken follows the reactive programming paradigm from the deepest Java service to the highest frontend component.
Reactive programming is a programming paradigm designed to maintain overall coherence by propagating the modifications of a reactive source (modification of a variable, user input, etc.) to the elements dependent on this source.
To do this, it uses the following stack:
Angular
Angular
Having the experience of creating OctoPerf AngularJS frontend, the simplest part for me was certainly to switch to Angular.
We already use TypeScript for OctoPerf, so the main difference is that Angular relies on RxJS from server-side communication with the Http client to the reactive forms.
Compared to AngularJS, it is a big improvement in terms of usability and maintainability.
That being said, I don’t feel like switching to Angular for OctoPerf yet.
Some features like tree views are not as complete. Also, we use Bootstrap, and I’m not sure how we could make it get along with Angular Material.
Server-Sent Events
The most challenging part here is to make Spring WebFlux return data according to the SSE specifications and call it from Angular.
But once working, it allows propagating modifications made on the file system or command execution logs up to the UI.
I’ll get back to that in another post blog.
Spring
Kraken Architecture
- First I started by creating a second frontend (I had to split the existing project into libraries),
- Then I created a
kraken-analysis
Java backend that wraps calls to the two other Java backends and to the analysis stack,
- I also configured a NGinx server
kraken-static
to serve Gatling’s report and debug responses as HTML content,
- Finally I added the load testing analysis stack:
kraken-influxdb
to store the metrics,
kraken-grafana
to display performance dashboards,
kraken-analysis-telegraf
to push monitoring metrics about the Docker host machine.
I also created several other Docker images that are only started when needed: kraken-gatling-recorder
to record HAR, kraken-gatling-debugger
to debug and kraken-gatling-runner
to execute load tests.
Automate everything
As the number of micro-services grows, the development process becomes a bit annoying if you need to manually test every modification:
While everything is packaged in Docker images and started automatically thanks to a docker-compose.yml
script in production, you have to start each required service by hand before doing your test using a dev environment.
For example, to test the Kraken Gatling frontend, you need to start the Executor backend, the Storage backend, the Analysis backend, the Static backend, InfluxDB, Grafana, etc.
To ease this process, we have the habit to use makefiles to automate everything.
So running an application is usually done with one single command like make serve-storage
.
Unit testing
Also, we heavily rely on Unit Tests to have good code quality.
This also ensures that any modification made to the backend applications works as expected without having to start the frontend and test it by hand.
Even though I did not achieve the same level of code coverage on Kraken than what we did on OctoPerf, it’s still pretty good:
Open Source
Kraken’s code is Open Source and available at https://github.com/OctoPerf/kraken.
If you want to know more about the Docker management application, please check out the Kraken Administration documentation page.
To learn more about a simplified Gatling load testing experience, please read the Kraken Gatling documentation page.
Conclusion
What started as a proof of concept is now an application.
I did not intend to create Kraken while starting to test new technologies, but one thing leading to another, here it is!
I will surely continue to develop Kraken as a side-project to OctoPerf.
The good thing with Kraken is that is can be used to run any tool provided:
- It uses the file system to store its data,
- It can be run inside one or several Docker container(s).
It’s even easier if it pushes metrics to InfluxDB, you can quickly get a nice Dashboard in Grafana.
Some other technical blog posts will be coming soon:
Your feedback is welcome, please leave a comment on this blog post or open a issue.