Unit 9 — Consuming Secrets in Kubernetes

Manages secrets on Kubernetes using Vault. The primary utility of the Operator lies in its ability to facilitate the consumption of secrets as native Kubernetes secrets. The Operator renders secrets natively as Kubernetes Secrets. The Operator also provides full secrets lifecycle management, including static and dynamic secrets from the Vault server. The Operator is also capable of managing the rotation of secrets, and notifying applications via rolling updates of a Deployment.

Supports a variety of authentication methods.

Designed to inject Vault secrets into your applications without requiring the applications to interact with Vault directly. The Agent Sidecar Injector accomplishes this by rendering Vault secrets to shared memory volumes. This approach offers remarkable convenience by allowing containers within a pod to consume secrets without being made Vault-aware.

Functions as a Kubernetes mutating webhook service, intercepting pod events and injecting Vault Agent container(s) into the pod if specific annotations exist on the resource.

Part of the vault-k8s project and can be installed with the Vault Helm chart.

Supports all authentication methods available to Vault Agent.

Provides secrets to applications running within Kubernetes, operating as a daemonset as opposed to a sidecar within each pod. Utilizes standard Kubernetes CSI Secrets Store volumes.

A user creates a SecretProviderClass object which specifies vault as the secret provider and contains the auth details and secrets to retrieve from Vault. When a Pod is created that references that SecretProviderClass, the Vault CSI provider uses that information along with the pod’s service account to retrieve and mount secrets into the pod’s CSI volume.

This is a blocking operation, as secrets must be retrieved and populated during the ContainerCreation phase, so a pod will not start until the secret has been read and written to the volume.

Supports Kubernetes and JWT authentication methods.

Scenario 1 → CSI. The platform team manages an application on an edge appliance that demands substantial uptime. This application is capable of reading secrets from a CSI volume and can fetch updates without necessitating a pod restart. A sidecar proves to be too resource intensive, and the appliance does not contain multiple Kubernetes nodes so a rolling update is not an option. Ultimately the team settles on the CSI driver for the ability to keep their application up when secrets are updated.

Scenario 2 → VSO. A platform team wants to leverage multiple Vault instances, and requires the ability to connect to multiple namespaces to retrieve secrets. They also need to leverage their AWS credentials to authenticate to Vault. The team decides quickly on VSO for its ability to render as native Kubernetes secrets, perform rolling updates, and leverage AWS credentials.

The discriminators: native Kubernetes Secrets and rolling updates → VSO. Vault-unaware containers, shared memory, no CRDs → Sidecar Injector. Per-node efficiency and no pod restart on update, at the cost of blocking startup and limited auth methods → CSI.

vault auth enable kubernetes

vault write auth/kubernetes/config \
  token_reviewer_jwt="<your reviewer service account JWT>" \
  kubernetes_host=https://192.168.99.100:<port> \
  kubernetes_ca_cert=@ca.crt

vault write auth/kubernetes/role/demo \
  bound_service_account_names=myapp \
  bound_service_account_namespaces=default \
  policies=default \
  audience=myapp \
  ttl=1h

The Kubernetes auth method validates service account JWTs and verifies their existence with the Kubernetes TokenReview API.

Parameters worth knowing:

• alias_name_source defaults to serviceaccount_uid. When you specify serviceaccount_uid, Vault uses a machine generated UID from the service account as the identity alias name. Using a service account UID is both the default and the recommended method as it is the more secure option. The tradeoff: a recreated service account with the same name gets a new UID and does not inherit the old identity — which surprises teams rebuilding namespaces via GitOps.
• bound_service_account_namespace_selector — a label selector for Kubernetes namespaces allowed to access this role. Accepts either a JSON or YAML object. Currently, label selectors with matchExpressions are not supported. To use label selectors, Vault must have permission to read namespaces on the Kubernetes cluster. If set with bound_service_account_namespaces, the conditions are ORed. This is how you avoid enumerating hundreds of namespaces.

Version note: starting in Kubernetes version 1.21, the BoundServiceAccountTokenVolume feature defaults to enabled. This changes the JWT token mounted into containers in two ways that are important for Kubernetes auth: it has an expiry time and is bound to the lifetime of the pod and service account, and the value of the JWT’s iss claim depends on the cluster’s configuration. The changes to token lifetime are important when configuring the token_reviewer_jwt option.

Vault can also use the client’s token if it is granted the system:auth-delegator cluster role in Kubernetes.