Terraform, Secrets, and Continuous integration

Terraform is about managing a desired state, and an observed state so changes can be expressed in a descriptive way and be applied on the required changes only. Any changes modifies a big JSON file representing the observed state and it's recommended to keep track of that file, e.g. we use Git.

It's all nice and warm until you start to manage resources to are persisting sensitive values in that big JSON state. Secrets in Git are a big no-no and you should have tools like git-secrets set up. Here are some solutions for you.

Storing the state, elsewhere, e.g. in an encrypted S3 object. The Dynamo integration offers a neat way to prevent race conditions between people of a same team.

Alternatively it’s possible to cipher the JSON file and keep it stored locally, which we will demonstrate here.

EJSON

Of the alternatives we've tried, EJSON from Shopify seemed to be the simplest one. It uses Curve25519 via golang.org/x/crypto/nacl/box to cipher each string value.

It's as easy as creating the pair of keys.

The public key is put inside the JSON document

Then running ejson encrypt on that file will cypher it in place. Meaning it’s idempotent. All the keys will remain clear while the values will be ciphered. Hence you can still manipulate that file as a JSON one.

We also put the public key into the GitLab CI variables, as unprotected. The private key is put as protected and masked.

The goal with Terraform is to amend the _public_key entry to the big JSON file, and store that into Git. When you want to use the state and its secrets, decrypt it back.

Continuous Integration

Some folks are using Atlantis and I'd love to. In the meantime we've build the poor's man version using GitLab CI. It's not that complex after all.

Some actions don't require any state to function. They are the bread and butter of the continuous integration. The following stage checks the formatting of the .tf files and validates the resources according to the plugins.

Being able to plan, requires the state to be present. Hence me must decrypt it beforehand. The before_script and after_script stages will replace the .ejson file with its usable counterpart.

And finally the tricky one, the apply phase. We require a manual action for it to force a human to review the plan. Or at least pretend to. Also, we add a twist to the plan phase and keep the plan as an artifact. The goal is to ensure that the apply matches what was reviewed during the previous phase.

Doing the git committing and pushing the changes back to GitLab using a deploy key is left as an exercise to the reader.

Recap

With a bunch of tools like ejson, to encrypt string inside a JSON file, and jq, the ultimate swissknife when it comes to JSON documents, it's not that hard of a task to use GitLab CI features to review, apply, and keep an history. I bet Terraform Enterprise and Atlantis are great. You can go quite far already with simple tools and some ingenuity.

At the company we've set up this for, having GitLab CI brought a new hammer to the toolbox. Some people are loving this new hammer and believe that it is part of the new solution to all current and future problems. My opinion is that each tool has its niche and should be used for that. E.g. do not deploy to Kubernetes using Terraform. Keep it for states that aren't meant to change every other day or are only accessible via a graphical user interface (GUI).

Senior developer