Terraform, Secrets, and Continuous integration

Yoan Blanc
4 min readApr 19, 2020

Terraform is about managing a desired state, and an observed state so changes can be expressed in a descriptive way and be applied on the required changes only. Any changes modifies a big JSON file representing the observed state and it's recommended to keep track of that file, e.g. we use Git.

It's all nice and warm until you start to manage resources to are persisting sensitive values in that big JSON state. Secrets in Git are a big no-no and you should have tools like git-secrets set up. Here are some solutions for you.

Storing the state, elsewhere, e.g. in an encrypted S3 object. The Dynamo integration offers a neat way to prevent race conditions between people of a same team.

Alternatively it’s possible to cipher the JSON file and keep it stored locally, which we will demonstrate here.


Of the alternatives we've tried, EJSON from Shopify seemed to be the simplest one. It uses Curve25519 via golang.org/x/crypto/nacl/box to cipher each string value.

It's as easy as creating the pair of keys.

$ ejson keygen
Public Key:
Private Key:

The public key is put inside the JSON document

"_public_key": "ce214db0ed7d28c8ab37bea3b95aedb77205b3f8df9cd10c7fbfa63f74f16d0c",
"secret": "s3cr€t5"

Then running ejson encrypt on that file will cypher it in place. Meaning it’s idempotent. All the keys will remain clear while the values will be ciphered. Hence you can still manipulate that file as a JSON one.

$ ejson encrypt data.ejson

We also put the public key into the GitLab CI variables, as unprotected. The private key is put as protected and masked.

The goal with Terraform is to amend the _public_key entry to the big JSON file, and store that into Git. When you want to use the state and its secrets, decrypt it back.

Continuous Integration

Some folks are using Atlantis and I'd love to. In the meantime we've build the poor's man version using GitLab CI. It's not that complex after all.

Some actions don't require any state to function. They are the bread and butter of the continuous integration. The following stage checks the formatting of the .tf files and validates the resources according to the plugins.

terraform validate:
stage: lint
- terraform fmt -check -recursive || (terraform fmt -diff; exit 1)
- terraform init -input=false
- terraform validate

Being able to plan, requires the state to be present. Hence me must decrypt it beforehand. The before_script and after_script stages will replace the .ejson file with its usable counterpart.


- |-
if [ -n "$EJSON_PRIVATE_KEY" ]
ejson decrypt terraform.tfstate.ejson > terraform.tfstate
echo ".-------------------------------."
echo "| The state is encrypted. |"
echo "| Provide the EJSON_PRIVATE_KEY |"
echo "`-------------------------------'"

- rm -rf $EJSON_KEYDIR terraform.tfstate

terraform plan:
stage: plan
- terraform init -input=false
- terraform plan

And finally the tricky one, the apply phase. We require a manual action for it to force a human to review the plan. Or at least pretend to. Also, we add a twist to the plan phase and keep the plan as an artifact. The goal is to ensure that the apply matches what was reviewed during the previous phase.

terraform plan:
stage: plan
- terraform init -input=false
- terraform plan
- >-
echo "{
\"_public_key\": \"$EJSON_PUBLIC_KEY\",
\"tfplan\": \"$(base64 -w0 tfplan)\"
}" > tfplan.ejson
- ejson encrypt tfplan.ejson
- tfplan.ejson

terraform apply:
stage: apply
when: manual
- plan
- master
- ejson decrypt tfplan.ejson
| jq ".tfplan"
| base64 -d > tfplan
- terraform apply -backup=- tfplan
- >-
cat terraform.tfstate
| jq ". + {_public_key: \"$EJSON_PUBLIC_KEY\"}"
> terraform.tfstate.ejson
- ejson encrypt terraform.tfstate.ejson
- git add terraform.tfstate.ejson

Doing the git committing and pushing the changes back to GitLab using a deploy key is left as an exercise to the reader.


With a bunch of tools like ejson, to encrypt string inside a JSON file, and jq, the ultimate swissknife when it comes to JSON documents, it's not that hard of a task to use GitLab CI features to review, apply, and keep an history. I bet Terraform Enterprise and Atlantis are great. You can go quite far already with simple tools and some ingenuity.

At the company we've set up this for, having GitLab CI brought a new hammer to the toolbox. Some people are loving this new hammer and believe that it is part of the new solution to all current and future problems. My opinion is that each tool has its niche and should be used for that. E.g. do not deploy to Kubernetes using Terraform. Keep it for states that aren't meant to change every other day or are only accessible via a graphical user interface (GUI).