Continuous Deployment on Nomad

The following article was presented during the Hashicorp Romandie User Group, June the 4th 2020. The goal was to show what was done with Nomad in the context of a bank with quite a lot of application to be migrated away from WebSphere. Most of them are relatively old and required first to be moved to GitLab. Developers are introduced to new practices, this is what is currently done.

The plan

The goals from day one were making things as simple as possible for the people as well as setting up some automation whenever possible. The breaking change from the current situation is to aim for the same workflow in any environment. Obviously, more eyes and checks are needed in the higher spheres.

Create a release from GitLab CI

One click to activate the mvn release:prepare step.

Ask for the deployment of a new release

A Merge Request describes the change, e.g. with the new version to be deployed.

 {
-   "version": "1.2.4"
+   "version": "1.3.0"
 }

There is no step three

Once the merge request is accepted and merged, the changes are live.

Continuous Integration

For some teams, the big change is moving from RTC (Rationale Team Concert) to Git; and using GitLab and its integrated continuous integration.

stages:
  - mvn verify

For Java projects, mvn verify does it all in 90% of the cases. It doesn’t have to be fancy. Most of the work has been hidden under the rug of the custom base image.

Pro tip: invest enough time into building those CI docker images.

Continous Delivery

The delivery is done by the GitLab CI but manually activated. So mvn release:prepare is manually triggered and mvn release:perform runs on the tag created by the former action.

stages:
  - mvn verify
  - mvn release:prepare
  - mvn release:perform

If this is the recommended setup, some folks are doing their own sauce with release branches. At the end of the day, it’s more YAML, more ways to get it wrong, and an harder recipe to run when it comes to do it locally.

Multiple deliveries?

The Java steps are producing Java artifacts, such as .war or .jar sent to Nexus (in our case but it could be Artifactory). For a while, we’ve also built the container image in the same projects which quickly felt a burden. In some cases, you’d want to upgrade all base layers without having to create new Java releases.

Pipeline trigger

A solution is to have another project for the Docker image, Using pipeline triggers and variables, when the Java release is done, the Docker image is built. We are then 100% sure that the same artifact is used in both.

$ curl -X POST \
       -F token=$TOKEN \
       -F ref=master \
       -F variables[artifactId]=myapp \
       -F variables[version]=$CI_COMMIT_TAG \
       $CI_API_V4_URL/projects/1234/trigger/pipeline

As most applications are from the same mold, it makes sense to unify how they where turned into container images. It is usually as easy as copy the .war, unzip the .war, remove the .war.

Continuous Deployment

Once you have a strong working CI, it tends to be used for everything. If it were to be compromised, that basket would hold all your eggs and nobody wants that.

Thus for the deployments, we’ve inverted the flow of control. The bot holds the credentials and listens to GitLab. Whenever changes are merged into master, they get deployed.

A simple bot was built using Python, python-gitlab, and subprocess. It would listen to changes, perform the deployments, and post a comment on the commit containing the changes.

Levant

James Rasell’s Levant enables some templating (just like Helm). This would alleviate the issue of some boiler-plate.

levant deploy -var-file config.yaml job.nomad

The config.yaml file holds the values, what needs to change most of the time.

version: 1.2.3
logging: INFO

And the job.nomad the skeleton of the deployment definition.

task {
  driver = "docker"
  config {
    image  = "hello/world:[[ .version ]]"
  }
  env {
    LOG = [[ .logging | toJSON ]]
  }
}

If you had the chance to play with Helm, you’ll realize that it’s much easier to template HCL than YAML. In fact YAML being fancy JSON, kustomize does it right, yet it’s way harder, go figure.

Levant alone worked for a while but wasn’t made to be integrated with GitLab the way we wanted it to be. So we’ve built our own tool.

Hello Bender!

We’ve taken Levant a step further by integrating its idea into GitLab CI. The whole project is about 500 lines of code. Definitively more than a week-end project, but a fun side-project. Nomad and go modules don’t play well together but things are improving.

It’s also written in Go, using it locally is also simple and fun.

On a Merge Request, one can ask bender to perform a job plan. The goal is to have a preview of what will change.

The the merge request is closed, the job is applied. The default Nomad policies (aka anonymous) permits everyone to see what’s going on, but not to interfere with the jobs themselves.

Et voilà!