![ApptioCareers-Employee[1]](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyMjUgMjc1IiB3aWR0aD0iMjI1IiBoZWlnaHQ9IjI3NSIgZGF0YS11PSJodHRwcyUzQSUyRiUyRnd3dy5hcHB0aW8uY29tJTJGd3AtY29udGVudCUyRnVwbG9hZHMlMkZlbGVtZW50b3IlMkZ0aHVtYnMlMkZBcHB0aW9DYXJlZXJzLUVtcGxveWVlMS1xZzVuenB5MHdoaWJtbzZvdmJqNGM3MHh0aTUzdmo1OHM0endwajh0dHMucG5nIiBkYXRhLXc9IjIyNSIgZGF0YS1oPSIyNzUiIGRhdGEtYmlwPSIiPjwvc3ZnPg== "ApptioCareers-Employee[1]")
IBM ApptioKubernetes DevOps Tools
Chapter 10: Kubernetes External Secrets
Like this article?
Subscribe to our LinkedIn newsletter to receive more educational content.
Kubernetes secrets are sensitive pieces of information that should be kept secure, such as passwords, API keys, and tokens. Without secrets, it is nearly impossible to use Kubernetes for most environments and workloads.
Backend applications might need access to the database for processing and storing data, frontend applications might require API keys to integrate with third-party APIs, and the Kubernetes cluster might need access to pull container images from private container registries.
While Kubernetes has built-in Secrets for storing and using sensitive data. However, it’s often insufficient because they are stored unencrypted in the Kubernetes data store. This article will show you how to securely install and use Kubernetes External Secrets for handling sensitive data in a Kubernetes cluster.
Secrets Management In Kubernetes
Secrets management in Kubernetes provides secure ways to manage the creation, storage, rotation, and removal of digital credentials while reducing human involvement and minimizing potential sources of error.
| Kubernetes Secrets Management Overview | |
|---|---|
| Concept | Summary |
| Kubernetes Secrets |
|
| Kubernetes External Secrets |
|
| External Secrets Operator (ESO) |
|
Kubernetes Secrets
Kubernetes Secrets are built-in objects that store and manage secrets in a Kubernetes cluster. Secrets are created and stored as a Kubernetes Secret object and used by pods or other objects in a cluster.
Kubernetes Secrets provide an easy and declarative way to create and manage secrets in a Kubernetes cluster.
However, there are a few downsides to using this built-in secrets management mechanism. Specifically, Kubernetes Secrets have several downsides such as:
- Stored as Base64 encoded objects so anyone with Cluster access can decode the secrets.
- Created either by kubectl CLI or in YAML manifests, making them insecure to integrate with version control systems.
- Difficult to manage and synchronize when managing multiple environments.
- No default mechanism to rotate and update the secrets
Kubernetes External Secrets
Kubernetes External Secrets store information outside the Kubernetes cluster while still allowing Kubernetes resources to use them. They are stored in an external service that Kubernetes interacts with to read and write secrets.
External secrets have several advantages over Kubernetes Secrets:
- Sensitive data is managed by external services outside the Kubernetes cluster and is less likely to be compromised as both systems need to be compromised to access sensitive data.
- Kubernetes External Secrets allow you to use different storage backends for your secrets, such as HashiCorp Vault, AWS Secrets Manager, Google Secrets Manager, or Azure Key Vault.
- Most importantly, you can take advantage of the security features of these services, such as encryption of data-at-rest and in-transit and scheduling for secrets’ rotation.
External Secrets Operator (ESO)
To use Kubernetes External Secrets, you must configure an external secrets backend and create a Kubernetes Secret object that points to the external backend. Kubernetes will then interact with the secret backend to read and write the secrets.
The External Secrets Operator makes using external secret management systems easier with Kubernetes. The External Secrets Operator will read the required information from the external API and inject it into a Kubernetes Secret for you. With this operator, you can easily incorporate secrets from providers like AWS Secrets Manager, HashiCorp Vault, and many more.
How to Manage Kubernetes External
Secrets with AWS Secrets Manager
To demonstrate Kubernetes External Secrets, we will use Amazon Elastic Kubernetes Service (EKS) as a Kubernetes Cluster and Amazon Secrets Manager as an external secret store.
To follow this tutorial, you’ll need:
- An AWS account
- awscli (Installed and configured with AWS credentials)
- kubectl
- jq
- eksctl (Optional, For deploying EKS cluster)
Deploying AWS EKS Cluster
To deploy an AWS EKS cluster, we will use eksctl CLI. To create the cluster, create a cluster.yaml manifest file with the below configuration.
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: eks-external-secrets
region: us-east-1
nodeGroups:
- name: ng-1
instanceType: t3.small
desiredCapacity: 1
volumeSize: 8
To apply the configuration, run this command in your terminal:
$ eksctl create cluster -f cluster.yaml
This will create an EKS cluster with one node group containing a single node in the us-east-1 AWS region. Once the cluster is ready, you should see output similar to the output below.
2022-09-05 18:47:47 [✔] EKS cluster "eks-external-secrets" in "us-east-1" region is ready.
Now, we can use this cluster and interact with it using kubectl. First, we must update the kubeconfig file with newly created cluster access. To update the kubeconfig, execute the below command.
$ aws eks --region us-east-1 update-kubeconfig --name eks-external-secrets
Test cluster access by running:
$ kubectl get pods
No resources found in default namespace.
Deploying External Secrets Operator
Install Helm to deploy the External Secrets Operator in the cluster. Once installed, verify theHelm version.
$ helm version --short
external-secrets by executing the below commands:
$ helm repo add external-secrets https://charts.external-secrets.io
$ helm install external-secrets \
external-secrets/external-secrets \
--namespace external-secrets \
--create-namespace \
--set installCRDs=true
Once installation is complete, you will see the below message:
external-secrets has been deployed successfully!
Configuring IAM roles for service accounts (IRSA) and Secrets Manager
With IAM roles for IRSA, you can map AWS IAM roles to Kubernetes Service Accounts. This feature allows you to manage AWS credentials for your applications running on EKS without managing static credentials directly.
To securely use External Secrets, we will use IRSA to provide AWS credentials. This way, the External Secrets pods will have access only to the secrets they need in our AWS Secrets Manager secret.
To use IRSA, the first step is to create an IAM OIDC Provider for your cluster (if one doesn’t already exist). After you associate the provider with your EKS cluster, you can create an IRSA service account that External Secrets will use.
To enable IAM OIDC Provider for your cluster, run the following command:
$ eksctl utils associate-iam-oidc-provider --cluster=eks-external-secrets --approve
secret-api-key and store the username and key as secret values.
$ SECRET_ARN=$(aws secretsmanager create-secret --name secret-api-key \
--secret-string "{\"username\":\"admin\",\"key\":\"SeCure@ApIKey\"}" \
--region us-east-1 | jq -r .ARN)
Next, create an IAM policy that will grant the permissions to interact with AWS Secrets Manager. For additional security, the following policy only allows for the description and retrieval of a single specified secret.
$ IAM_POLICY_ARN=$(aws iam create-policy --policy-name eks-external-secrets-reader --policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue"
],
"Resource": ["'${SECRET_ARN}'"]
}
]
}' | jq -r .Policy.Arn)
Finally, create an IRSA service account for the External Secrets Operator to authenticate and fetch the required secrets from AWS Secrets Manager.
Here we are creating this IRSA service account in the default namespace.
$ eksctl create iamserviceaccount \
--name external-secrets-irsa \
--namespace default \
--cluster eks-external-secrets \
--role-name "external-secrets-irsa-role" \
--attach-policy-arn $IAM_POLICY_ARN \
--approve \
--override-existing-serviceaccounts
To verify the successful creation of the Kubernetes Service Account, execute the following command:
$ kubectl get sa
NAME SECRETS AGE
default 1 39m
external-secrets-irsa 1 3s
Configuring External Secrets Operator
To work with External Secrets, we must create two custom Kubernetes resources. ExternalSecret and SecretStore.
ExternalSecret describes what data should be retrieved, processed, and stored as a Kubernetes Secret object.
SecretStore specifies how to access and interact with the external APIs. In this example, it’s AWS Secrets Manager.
First, create two manifest files, external-secret.yaml and secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: eks-external-secret
spec:
refreshInterval: 10m
secretStoreRef:
name: eks-secret-store
kind: SecretStore
target:
name: eks-secret
data:
- secretKey: username
remoteRef:
key: secret-api-key
property: username
- secretKey: key
remoteRef:
key: secret-api-key
property: key
external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: eks-secret-store
namespace: default
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-irsa
secret-store.yaml
default namespace because an IRSA service account was also created in the default namespace.
$ kubectl apply -f secret-store.yaml
$ kubectl apply -f external-secret.yaml
Verify ExternalSecret deployment status.
$ kubectl get externalsecrets.external-secrets.io
NAME STORE REFRESH INTERVAL STATUS READY
eks-external-secret eks-secret-store 10m SecretSynced True
Status as SecretSynced.
After you’ve created the ExternalSecret resource, you’ll be able to see the new Kubernetes Secret that has been synchronized with the Secrets Manager store. Execute the following command:
$ kubectl describe secret eks-secret
Name: eks-secret
Namespace: default
Labels: <none>
Annotations: reconcile.external-secrets.io/data-hash: a946eefe36be05541b6060cc5c7b467b
Type: Opaque
Data
====
key: 13 bytes
username: 5 bytes
You have now created a Kubernetes Secret that can be used within your pod specification for use by your applications.
Consuming Secret in Pod
By syncing your AWS Secrets Manager secret to a Kubernetes Secret, External Secrets allows you to use and consume the secret in your Pod specification.
We will deploy a simple busybox pod in the default namespace and use the secret via pod environment variables.
Create a manifest file external-secrets-demo-pod.yaml with the following specifications.
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- image: busybox:1.35.0
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
name: busybox
env:
- name: API_USERNAME
valueFrom:
secretKeyRef:
name: eks-secret
key: username
- name: API_KEY
valueFrom:
secretKeyRef:
name: eks-secret
key: key
restartPolicy: Always
To deploy the pod, run the following command:
$ kubectl apply -f external-secrets-demo-pod.yaml
Once the pod is in a running state, run the following commands to get the container’s shell.
$ kubectl exec -it busybox -- sh
From the container’s shell, you can use echo to print the environment variables to view the secrets.
/ # echo $API_USERNAME
admin
/ # echo $API_KEY
SeCure@ApIKey
Comparing Kubernetes Secrets and External Secrets
| Kubernetes External Secrets vs. Kubernetes Secrets | ||
|---|---|---|
| Feature | Kubernetes Secrets | Kubernetes External Secrets |
| Availability | Kubernetes’s default secret management | Need to install as Custom Resource Definition, and custom Operator |
| Secret Lifecycle Management | Difficult to manage secrets at a large scale and across multiple environments | Secrets are stored externally and inherit all the lifecycle features of external service |
| Security | Secrets are not stored as encrypted objects; any entity having cluster access can decode the secrets data | Secrets are managed outside of the cluster and stored encrypted |
| Version Control System | Kubernetes Secrets are not secure to include in the Version Control Systems | Most external secret management tools provide built-in version history. YAML manifests can be committed to version control systems as they only contain a reference to the secrets |
Deploying Kubernetes External Secrets in a Large-scale Production Environment
Deploying and managing secrets in a large-scale production environment requires awareness of and proactiveness against risks like data breaches. There are a few things to consider when deploying secrets in a production environment to ensure the security of your data and protect your systems from unauthorized access.
To mitigate the risk of secrets exposure, best practices include:
- Encrypt all sensitive data at rest and in transit
- Rotate secrets regularly
- Use strong passwords and passphrases for accessing external secrets backends
- Restrict access to secrets to only those who need it
- Monitor for unauthorized access using audit logs
Conclusion
In this article, we looked at the need for storing secrets securely in Kubernetes and explored External Secrets. While Kubernetes Secrets are a great way to store secrets within the cluster, they have security and lifecycle management limitations.
Kubernetes External Secrets address these shortcomings by storing secrets externally and providing an in-built secret rotation, robust lifecycle management, and multiple authentication mechanisms. We also reviewed the advantages of using External Secrets over standard Kubernetes Secrets.
If you are looking for a secure and robust way to manage secrets in your Kubernetes cluster, External Secrets are a great option to consider.
