Terraform, Secrets, and Continuous integration

Terraform is about managing a desired state, and an observed state so changes can be expressed in a descriptive way and be applied on the required changes only. Any changes modifies a big JSON file representing the observed state and it's recommended to keep track of that file, e.g. we use Git.

It's all nice and warm until you start to manage resources to are persisting sensitive values in that big JSON state. Secrets in Git are a big no-no and you should have tools like git-secrets set up. Here are some solutions for you.

Storing the state, elsewhere, e.g. in an encrypted S3 object. The Dynamo integration offers a neat way to prevent race conditions between people of a same team.

Alternatively it’s possible to cipher the JSON file and keep it stored locally, which we will demonstrate here.

EJSON

Of the alternatives we've tried, EJSON from Shopify seemed to be the simplest one. It uses Curve25519 via golang.org/x/crypto/nacl/box to cipher each string value.

It's as easy as creating the pair of keys.

$ ejson keygen
Public Key:
ce214db0ed7d28c8ab37bea3b95aedb77205b3f8df9cd10c7fbfa63f74f16d0c
Private Key:
50750f48c4c2cae3eb60a9c7e532e992749c0c4b54f5c2c8ab55b47ec40b4a10

The public key is put inside the JSON document

{
  "_public_key": "ce214db0ed7d28c8ab37bea3b95aedb77205b3f8df9cd10c7fbfa63f74f16d0c",
  "secret": "s3cr€t5"
}

Then running ejson encrypt on that file will cypher it in place. Meaning it’s idempotent. All the keys will remain clear while the values will be ciphered. Hence you can still manipulate that file as a JSON one.

$ ejson encrypt data.ejson

We also put the public key into the GitLab CI variables, as unprotected. The private key is put as protected and masked.

The goal with Terraform is to amend the _public_key entry to the big JSON file, and store that into Git. When you want to use the state and its secrets, decrypt it back.

Continuous Integration

Some folks are using Atlantis and I'd love to. In the meantime we've build the poor's man version using GitLab CI. It's not that complex after all.

Some actions don't require any state to function. They are the bread and butter of the continuous integration. The following stage checks the formatting of the .tf files and validates the resources according to the plugins.

terraform validate:
  stage: lint
  scripts:
    - terraform fmt -check -recursive || (terraform fmt -diff; exit 1)
    - terraform init -input=false
    - terraform validate

Being able to plan, requires the state to be present. Hence me must decrypt it beforehand. The before_script and after_script stages will replace the .ejson file with its usable counterpart.

variables:
  EJSON_KEYDIR: ".keys"

before_script:
  - |-
    if [ -n "$EJSON_PRIVATE_KEY" ]
    then
      echo -n $EJSON_PRIVATE_KEY > $EJSON_KEYDIR/$EJSON_PUBLIC_KEY
      ejson decrypt terraform.tfstate.ejson > terraform.tfstate
    else
      echo ".-------------------------------."
      echo "| The state is encrypted.       |"
      echo "| Provide the EJSON_PRIVATE_KEY |"
      echo "`-------------------------------'"
    fi

after_script:
  - rm -rf $EJSON_KEYDIR terraform.tfstate

terraform plan:
  stage: plan
  only:
    variables:
      - $EJSON_PRIVATE_KEY
  script:
    - terraform init -input=false
    - terraform plan

And finally the tricky one, the apply phase. We require a manual action for it to force a human to review the plan. Or at least pretend to. Also, we add a twist to the plan phase and keep the plan as an artifact. The goal is to ensure that the apply matches what was reviewed during the previous phase.

terraform plan:
  stage: plan
  only:
    variables:
      - $EJSON_PRIVATE_KEY
  script:
    - terraform init -input=false
    - terraform plan
      -out=tfplan
    - >-
      echo "{
      \"_public_key\": \"$EJSON_PUBLIC_KEY\",
      \"tfplan\": \"$(base64 -w0 tfplan)\"
      }" > tfplan.ejson
    - ejson encrypt tfplan.ejson
  artifacts:
    paths:
      - tfplan.ejson

terraform apply:
  stage: apply
  when: manual
  needs:
    - plan
  only:
    ref:
      - master
    variables:
      - $EJSON_PRIVATE_KEY
  script:
    - ejson decrypt tfplan.ejson
      | jq ".tfplan"
      | base64 -d > tfplan
    - terraform apply -backup=- tfplan
    - >-
      cat terraform.tfstate
      | jq ". + {_public_key: \"$EJSON_PUBLIC_KEY\"}"
      > terraform.tfstate.ejson
    - ejson encrypt terraform.tfstate.ejson
    - git add terraform.tfstate.ejson

Doing the git committing and pushing the changes back to GitLab using a deploy key is left as an exercise to the reader.

Recap

With a bunch of tools like ejson, to encrypt string inside a JSON file, and jq, the ultimate swissknife when it comes to JSON documents, it's not that hard of a task to use GitLab CI features to review, apply, and keep an history. I bet Terraform Enterprise and Atlantis are great. You can go quite far already with simple tools and some ingenuity.

At the company we've set up this for, having GitLab CI brought a new hammer to the toolbox. Some people are loving this new hammer and believe that it is part of the new solution to all current and future problems. My opinion is that each tool has its niche and should be used for that. E.g. do not deploy to Kubernetes using Terraform. Keep it for states that aren't meant to change every other day or are only accessible via a graphical user interface (GUI).