Manage your Kubernetes secrets with Hashicorp Vault

Once your Kubernetes clusters starts to grow, managing secrets can be a challenge. Yaml definitions of secrets are base64 encoded, so finding errors could be quite difficult. And yes, you could use a tool to convert these secrets from an env file, to a Kubernetes secret yaml definition using a simple python script for example. Maybe even have these stored in a S3 Bucket, but where’s the security in that?. If you’re serious about security in Kubernetes, you need a secret management tool that provides a single source of secrets, credentials, attaching security policies, etc. In other words, you need Hashicorp Vault.

This post outlines a process to use vault within Kubernetes to make the secret management more secure and robust with GoDaddys external secrets project. It was based on the talk that Seth Vargo gave at Cloud Next ‘18.

Prerequisites:

• A Kubernetes cluster (or more)

Tools used:

• Kubectl
• Vault CLI
• Helm v3

How to use Hashicorp Vault with Kubernetes?

I’ll divide this into 4 different sections

1. Run Vault on Kubernetes

The goal of this first section is to answer two questions:

1. Why would I run this on Kubernetes? Is it a good idea? Should I?
2. And how do I do it?

The first answer is YES… So I’ll just move on to the second question.

Vault as a service

Since we’re using Vault to manage our secrets, we’ll first need to install it in our Cluster.

I want to make it clear, and it is really important to understand that we want to think of Vault as a service. This means:

Vault is available at an IP/DNS address for consumers. Kubernetes is simply an implementation detail.

So yes, I’m running Vault under Kubernetes and I’m leveraging a lot of the functionality that Kubernetes provides. But ultimately, I’m delivering secrets management as a service. So there’s Vault, and an API and an IP/DNS address. And users ultimately shouldn’t care whether Vault is running on a VM or a container or bare metal. It doesn’t really matter. So we really want to think about this as Kubernetes is providing the building blocks that let us run Vault as a highly-available service.

For the deployment of the Vault cluster, I recommend using official Vault helm chart maintained by Hashicorp.

git clone https://github.com/hashicorp/vault-helm.git
cd vault-helm

For the deployment of mine cluster, I’ll be using an ingress based access to my service, but you could use a Load Balancer, obtaining its IP address and just that as your vault service endpoint. In my case, my endpoint will be https://vault.craftech.io

It is worth noticing that Hashicorps chart already implements some great features for the cluster deployment, such as HA using pod anti affinity settings, making Vault deploy its replicas into different nodes of the cluster providing real high availability.

After you finish customizing the values.yaml, you can install Vault:

kubectl create namespace vault
helm install vault . --namespace vault --values values.yaml

Once finished, you can initialize your Vault cluster running:

kubectl exec -ti --namespace vault vault-0 vault operator init

This will return the initial root token and unseal keys. For more information, go to Initializing the Vault.

And that’s pretty much it. We have a highly available Vault cluster running under Kubernetes accessible by an IP/DNS address.

We can use vaults CLI to check status, just by exporting some environment variables:

export VAULT_ADDR=https://vault.craftech.io
export VAULT_TOKEN=<initial root token>
export VAULT_CACERT=/path/to/ca.crt #only if your not using a well known CA, like Let's Encrypt.

Now we can check status running vault status:

$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.3.3
Cluster Name    vault-cluster-dbf95b73
Cluster ID      dc129ce4-5566-f4c9-23a5f-422c2febedfc
HA Enabled      true
HA Cluster      http://10.33.0.7:8201k
HA Mode         active

2. Deploying and Configuring External Secrets in Kubernetes

Kubernetes External Secrets allows you to use external secret management systems, like AWS Secrets Manager or HashiCorp Vault, to securely add secrets in Kubernetes.

This is achieved by extending the Kubernetes API by adding a ExternalSecrets object using Custom Resource Definition and a controller to implement the behavior of the object itself.

We need to edit the values.yaml file found in the repo. There are several configurations to set up, depending if your using AWS Secrets Manager, Azure Key Vault, etc. We are focusing on Vault, so the important parts in the values.yaml are the VAULT_ADDR and allow creation of a serviceaccount for the service itself (we are then going to use that serviceaccount to authenticate against Vault).

Deploying is again done by the use of helm, but this time you add a helm repo to your helm source lists:

helm repo add external-secrets https://godaddy.github.io/kubernetes-external-secrets/
kubectl create namespace external-secrets
helm install kubernetes-external-secrets external-secrets/kubernetes-external-secrets --namespace external-secrets --values values.yaml

Now let’s move on to the most anticipated part, creating and managing Kubernetes secrets with Vault.

3. Connect to Vault from Kubernetes

Authentication in Vault is the process which users or machines supply information to Vault. Vault verifies that information either with internal data or some external third party, and then, assuming that is valid, converts that into a token with policy authorization attach to it.

Mental Reset

You have to forget everything. Forget everything you’ve just read, because the next section is: Now that I have Vault running, how do I connect to it from Kubernetes?

But I want you to forget that Vault is running in Kubernetes, instead Vault is just a thing running, available at an IP/DNS address. Everything described in the next section will work with any Kubernetes cluster, deployed on any cloud or on-prem infrastructure.

In a more graphical way:

1. We can see the Vault server in the middle which would receive a request from a user, machine or application (in this case, the ExternalSecrets’ serviceaccount).
2. Then, Vault verifies that information with the Kubernetes Token Reviewer, verifying the serviceaccounts JWT token validity, timestamp, etc. Assuming that’s successful, it goes back to Vault saying “Yup, this looks great”.
3. After that, Vault goes back to its own internal policy engine saying, well, the things in this particular namespace, get this particular policies attach to it… “here’s a vault token with that policy attach to it”.
4. Finally, Vault returns the token to ExternalSecrets’ serviceaccount, which would then be used for all future requests.

NOTE: The pattern Vault uses to authenticate Pods depends on sharing the JWT token over the network. Given the security model of Vault, this is allowable because Vault is part of the trusted compute base. In general, Kubernetes applications should not share this JWT with other applications, as it allows API calls to be made on behalf of the Pod and can result in unintended access being granted to 3rd parties.

But as the great Linus Torvald once

“Talk is cheap. Show me the code.” — Linus Torvalds

So, how do we actually connect multiple Kubernetes clusters to my Vault SaaS?

Well, I reduced the connection based on a couple of commands:

1 Export in environment variables the name of your cluster and Vault connection details:

export CLUSTER_NAME=my-cluster
export VAULT_ADDR=https://vault.craftech.io
export VAULT_TOKEN=9zW0XxOs.eoi3948yn #Initial root token

As an example, I’ll be using an EKS cluster, so my cluster name is:

export CLUSTER_ID=arn:aws:eks:us-east-1:<aws-account-ID>:cluster/my-cluster

2. Apply a clusterrolebinding for the ExternalSecrets’ serviceaccount with the role auth-delegator.

kubectl apply \
  --filename=-<<EOH
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: kubernetes-external-secrets
  namespace: external-secrets
EOH

3. Create a policy for the serviceaccounts token to be attached.

vault policy write ${CLUSTER_NAME}-kv-rw - <<EOH
path "kv/data/${CLUSTER_NAME}/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
EOH

4. Saved in environment variables cluster configuration and ExternalSecrets serviceaccounts JWT token.

SECRET_NAME="$(kubectl get serviceaccount kubernetes-external-secrets -o go-template='{{ (index .secrets 0).name }}')"
SERVICEACCOUNT_TOKEN="$(kubectl get secret ${SECRET_NAME} \
  -o go-template='{{ .data.token }}' | base64 --decode)"
K8S_HOST="$(kubectl config view --raw \
  -o go-template="{{ range .clusters }}{{ if eq .name \"${CLUSTER_ID}\" }}{{ index .cluster \"server\" }}{{ end }}{{ end }}")"
K8S_CACERT="$(kubectl config view --raw \
  -o go-template="{{ range .clusters }}{{ if eq .name \"${CLUSTER_ID}\" }}{{ index .cluster \"certificate-authority-data\" }}{{ end }}{{ end }}" | base64 --decode)"

5. Enable the Kubernetes auth method for this particular cluster using path

vault auth enable --path="${CLUSTER_NAME}" kubernetes

6. Configure the clusters auth method in Vault.

vault write auth/${CLUSTER_NAME}/config \
  kubernetes_host="${K8S_HOST}" \
  kubernetes_ca_cert="${K8S_CACERT}" \
  token_reviewer_jwt="${SERVICEACCOUNT_TOKEN}"

7. Finally, create a role in the Kubernetes auth method to be used by the ExternalSecrets’ serviceaccount:

vault write auth/${CLUSTER_NAME}/role/${CLUSTER_NAME}-role \
  bound_service_account_names="kubernetes-external-secrets" \
  bound_service_account_namespaces="external-secrets" \
  policies="default,${CLUSTER_NAME}-kv-rw" \
  ttl="15m"

And that’s it! We have our Kubernetes cluster configured to create secrets based on externalsecrets definitions.

4. Creating a secret from an external secret

Finally, we reach the part that would become the most anticipated: actually creating a secret in Kubernetes obtained from a Vault’s secret.

In order to do that, we’ll have to create an ExternalSecret definition. This can be achieved from a yaml definition like this one:

kubectl apply \
  --filename=-<<EOH
---
apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: test
spec:
  backendType: vault
  vaultMountPoint: my-cluster #created at step 5
  vaultRole: my-cluster-role #created at step 7
  dataFrom:
    - kv/data/test/test
EOH

And we get:

externalsecret.kubernetes-client.io/test created

According to GoDaddy’s Github documentation, if you use Vault:

Vault values are matched individually. If you have several keys in your Vault secret, you will need to add them all separately

I opened up an issue to them, explaining that Vault actually admins just referencing the secret location to pull every key-value stored in it. At the time of writing this post, I haven’t got an answer, yet.

We can obtain the external secrets created with:

$ kubectl get externalsecrets
NAME   LAST SYNC   STATUS              AGE
test   5s          ERROR, Status 404   46s

This shows 404 Not Found, because the secret doesn’t exist in Vault… yet.

So now we’ll move on to Vault’s Web UI, and create the secret:

We save it, and check again:

$ kubectl get externalsecrets
NAME   LAST SYNC   STATUS    AGE
test   3s          SUCCESS   13m

Bingo! It appears to be working. Let’s check the Kubernetes secret created:

Conclusion

We have provisioned a Kubernetes cluster with secrets obtained form a Vault cluster running as a service, avoiding the need to encrypt our secrets ourself or stored them in different places.

We also benefit from being able to visualize the changes of secrets in plain text from a web ui, so troubleshooting on secrets become an easy task.

Questions?

Please feel free to join us on Craftech’s community Slack and ask any questions.