Local Vault Enterprise + Kubernetes Lab (macOS / Podman)

Check these are installed. If not, install via Homebrew.

brew install podman minikube kubectl helm

Verify versions:

podman --version
minikube version
kubectl version --client
helm version

You don’t need exact versions matched — anything reasonably recent (Podman 5.x, minikube 1.33+, kubectl 1.29+, Helm 3.x) works fine for this lab.

Step 1.0 — Check your actual physical RAM before setting machine memory

Podman’s memory cap on macOS isn’t arbitrary — it won’t let you allocate more RAM to the VM than the host physically has, minus a reserve for macOS itself. If Podman refuses your requested --memory value and reports a lower cap instead (a real example from testing this guide: requesting 8192 MB on an 8GB Mac got capped to 7904 MB), check what you’re actually working with:

sysctl hw.memsize

This prints bytes. Divide by 1,048,576 to get MB — e.g. 8589934592 = 8192 MB total.

• If the cap Podman reports is within a few hundred MB of your total physical RAM: that’s the real ceiling, not a bug — macOS needs headroom too. A 3-node HA Vault cluster plus Minikube’s own overhead will be tight but workable on 8GB if you keep resource requests low (this guide does). On more RAM, you’ll have more comfortable margin.
• If Podman’s cap is far below your total physical RAM (e.g. you have 16GB+ but it’s still capping around 8GB): the machine was likely initialized once with a low value and Podman is holding onto that. Destroy and recreate it explicitly (Step 1.1 below).

Step 1.1 — Initialize the Podman machine

Podman on macOS isn’t native — it runs a lightweight Linux VM under the hood (via qemu on Apple hardware). You need that VM running before minikube can use podman as a driver.

If you already have a machine from before and need to resize it:

podman machine stop
podman machine rm

Then initialize with an explicit memory value based on what Step 1.0 told you — leave roughly 500MB–1GB of your total physical RAM for macOS, and use the rest:

podman machine init --cpus 4 --memory <YOUR_VALUE> --disk-size 40

Concretely: on an 8GB Mac, that’s around 7000–7900. On 16GB, around 14000–15000. Don’t copy a number from someone else’s run — compute it from your own sysctl hw.memsize output.

Step 1.2 — Set the machine to rootful mode

This is the step almost everyone misses, and it’s the #1 cause of minikube+podman failures on macOS.

By default, newer Podman versions create the machine in rootless mode. Minikube’s podman driver needs rootful — it requires privileges to manage networking and cgroups that rootless mode restricts.

podman machine set --rootful

Step 1.3 — Start the machine

podman machine start

Verify it’s actually rootful:

podman machine list

You should see a * next to your default machine and it should show as running. Then confirm rootful mode:

podman info --format '{{.Host.Security.Rootless}}'

This should print false. If it prints true, Step 1.2 didn’t take — stop the machine (podman machine stop), re-run Step 1.2, and start it again.

Step 1.4 — Sanity check Podman itself

podman run --rm hello-world

If this pulls the image and prints the “Hello from Docker!” (yes, that’s the actual message, it’s the same test image) confirmation, Podman is working correctly. Don’t proceed to minikube until this succeeds.

Step 1.5 — Start minikube

minikube start --driver=podman --container-runtime=containerd --cpus=4 --memory=<VALUE>

Leave the podman machine’s own overhead some room — set this to roughly 800MB–1GB below the --memory value you gave podman machine init in Step 1.1, not equal to it. E.g. if your machine got 7900 MB, use --memory=7000 here. That headroom covers the VM’s own processes (podman’s API layer, networking, etc.), which matters once you’re running a 3-node Vault cluster instead of a single pod.

Two flags worth understanding, not just copying:

• --driver=podman — tells minikube to create its “node” as a Podman container instead of a VM or Docker container.
• --container-runtime=containerd — this is the container runtime running inside the minikube node itself (a separate concern from the driver). Use containerd here rather than the default docker — running a Docker daemon nested inside a Podman container has historically been flaky. containerd avoids that nesting problem entirely.

This will take a few minutes on first run (pulling the minikube node image).

Step 1.6 — Verify the cluster

kubectl get nodes

Expected output: one node, status Ready.

kubectl get pods -A

Expected: a handful of system pods (coredns, kube-proxy, storage-provisioner, etc.) all Running.

If you want a visual, optionally:

minikube dashboard

This opens a browser-based cluster view. Not required for anything downstream — skip if you’d rather stay on the CLI.

Phase 1 troubleshooting

This replaces dev mode with an actual 3-node cluster: unseal keys, root token generated at init, Raft integrated storage, nodes joining as peers, one active node and two standbys — the real operational shape, not the single-binary shortcut. The goal is still the same trust chain as before (ServiceAccount token → Vault token, scoped by policy), but now you’re building it against infrastructure that behaves like the real thing, including a step you can’t skip in dev mode: initializing and unsealing.

Step 2.1 — Add the HashiCorp Helm repo

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Step 2.2 — Get your Enterprise license and stage it as a Secret

You’ll have a license file (.hclic) or a license string from HashiCorp. Create the namespace and store it as a Kubernetes Secret — the Helm chart references this to activate Enterprise features on boot.

kubectl create namespace vault

kubectl create secret generic vault-ent-license \
  --namespace vault \
  --from-literal=license="$(cat /path/to/your/vault.hclic)"

Verify the secret actually got real content before moving on. If the path above is wrong, cat fails but the shell doesn’t stop — you’ll end up with a secret that “created successfully” but is empty, and the failure won’t surface until Vault refuses the license much later. Check now:

kubectl get secret vault-ent-license -n vault -o jsonpath='{.data.license}' | base64 -d | head -c 50

This should print the start of your actual license content. If it prints nothing, fix the path and recreate the secret before continuing.

Check what modules your license actually includes before going further. If you’re requesting this from an internal HashiCorp licensing portal, make sure you get a standard Platform Standard license, not one with a pki-only add-on attached. A pki-only license locks the entire cluster into PKI-only mode — every secrets engine except PKI gets rejected with a 400, which will break the KV setup in Step 2.9 and everything in Phase 3A/3B. You can check after the fact with vault license inspect, but it’s faster to just confirm at request time.

Step 2.3 — Install Vault in HA mode with Raft storage

Before running this: check the Vault Enterprise Docker Hub tags for the current version. 1.17.6-ent below is a placeholder from when this guide was written — using a stale tag isn’t fatal, but you’ll want to know what version you’re actually running rather than inherit one by accident.

helm install vault hashicorp/vault \
  --namespace vault \
  --set 'server.image.repository=hashicorp/vault-enterprise' \
  --set 'server.image.tag=1.17.6-ent' \
  --set 'server.enterpriseLicense.secretName=vault-ent-license' \
  --set 'server.ha.enabled=true' \
  --set 'server.ha.replicas=3' \
  --set 'server.ha.raft.enabled=true' \
  --set 'server.ha.raft.setNodeId=true' \
  --set 'server.affinity=' \
  --set 'injector.affinity='

The server.affinity= line matters more than it looks. The chart ships a hard pod anti-affinity rule by default: it refuses to schedule two Vault server pods on the same Kubernetes node, which is correct behavior for a real multi-node cluster. Minikube is single-node. Without this override, vault-0 schedules fine (first pod, nothing to conflict with), and vault-1/vault-2 sit in Pending forever — you’ll see pod does not have a host assigned if you try to exec into them. Setting server.affinity= to empty removes the rule entirely, which is the right call for a single-node lab and the wrong call for production.

injector.affinity= is the same fix applied to the Agent Injector deployment, which ships the identical default rule. It’s easy to miss because the injector runs as a single replica — the rule only bites when Kubernetes needs to schedule a second injector pod alongside the first, which doesn’t happen on day one. It happens the first time the injector pod gets replaced (a kubectl rollout restart, a helm upgrade, a node reboot, or recovering from the TLS issue in the Phase 3A troubleshooting table below): the new pod can’t schedule while the old one holds the node, the old one won’t terminate until the new one is ready, and you get a permanent Pending deadlock. Setting it now avoids ever hitting that.

What changed from dev mode, mechanically: no more single ephemeral pod. This creates a StatefulSet — vault-0, vault-1, vault-2 — each with its own persistent volume claim (backed by minikube’s default storage-provisioner addon, already active from Phase 1). None of them are usable yet.

Step 2.4 — Watch the pods come up sealed

kubectl get pods -n vault

Expected: all three pods Running but showing 0/1 ready. That 0/1 is correct and expected — a sealed, uninitialized Vault fails its readiness probe on purpose. Don’t troubleshoot this; it’s the normal starting state.

Step 2.5 — Initialize vault-0

This is the step that doesn’t exist in dev mode. Initializing generates the root key, splits it into unseal key shares, and issues the initial root token — this only happens once, against exactly one node.

kubectl exec -n vault -it vault-0 -- vault operator init -key-shares=1 -key-threshold=1

-key-shares=1 -key-threshold=1 is a lab simplification — one unseal key, no splitting. Never do this in production; real deployments use -key-shares=5 -key-threshold=3 (or similar) so no single person holds the whole key. For this lab, it just makes the walkthrough shorter.

Save the two values this prints — you’ll use both repeatedly:

• Unseal Key 1
• Initial Root Token

Step 2.6 — Unseal vault-0

kubectl exec -n vault -it vault-0 -- vault operator unseal <unseal-key-from-2.5>

Confirm:

kubectl exec -n vault -it vault-0 -- vault status

Expected: Sealed: false, HA Mode: active. vault-0 is now your cluster leader.

Step 2.7 — Join vault-1 and vault-2 as Raft peers

Each standby has to explicitly join the Raft cluster, then get unsealed with the same unseal key from Step 2.5 (the key material is shared across the cluster once a node joins — it’s not a separate key per node).

kubectl exec -n vault -it vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
kubectl exec -n vault -it vault-1 -- vault operator unseal <unseal-key-from-2.5>

kubectl exec -n vault -it vault-2 -- vault operator raft join http://vault-0.vault-internal:8200
kubectl exec -n vault -it vault-2 -- vault operator unseal <unseal-key-from-2.5>

vault-0.vault-internal is the per-pod DNS name the chart’s headless service provides — this is how Raft peers find each other inside the cluster.

Step 2.8 — Verify the cluster is healthy

kubectl get pods -n vault

Expected: all three pods 1/1 Running.

Log in so your CLI session is authenticated (this writes a token file inside the pod’s filesystem, so it persists across separate kubectl exec calls for the life of the pod):

kubectl exec -n vault -it vault-0 -- vault login <root-token-from-2.5>

Check the Raft peer set:

kubectl exec -n vault -it vault-0 -- vault operator raft list-peers

Expected: three rows, one leader and two follower/voters, all true under voter.

Step 2.9 — Enable the KV v2 secrets engine

Unlike dev mode, a freshly initialized Vault has nothing mounted at secret/ — you have to enable it explicitly:

kubectl exec -n vault -it vault-0 -- vault secrets enable -path=secret kv-v2

Step 2.10 — Enable the Kubernetes auth method

kubectl exec -n vault -it vault-0 -- vault auth enable kubernetes

Step 2.11 — Configure the Kubernetes auth method

This tells Vault how to reach the Kubernetes API to validate ServiceAccount tokens it’s handed.

kubectl exec -n vault -it vault-0 -- sh -c \
  'vault write auth/kubernetes/config \
    kubernetes_host="https://${KUBERNETES_PORT_443_TCP_ADDR}:443"'

KUBERNETES_PORT_443_TCP_ADDR is an environment variable Kubernetes automatically injects into every pod — it points at the internal kubernetes API service. You didn’t set it; it’s just there.

Note what you’re not doing: manually passing a CA cert or a reviewer JWT. Vault 1.9+ auto-detects both from its own pod’s mounted ServiceAccount, since that pod already has a token and CA cert available at /var/run/secrets/kubernetes.io/serviceaccount/. You only override this if Vault needs to validate tokens using a different ServiceAccount than its own — not needed here.

Step 2.12 — Write a test secret

kubectl exec -n vault -it vault-0 -- vault kv put secret/myapp/config username=demo password=demo123

Step 2.13 — Write a policy scoped to that secret path

kubectl exec -n vault -it vault-0 -- sh -c 'vault policy write myapp-read - <<EOF
path "secret/data/myapp/*" {
  capabilities = ["read"]
}
EOF'

Note the path: KV v2 prefixes actual data paths with data/ internally (metadata, versioning, etc. live alongside it) — this trips people up constantly. secret/myapp/config (what you wrote to) becomes secret/data/myapp/config (what you grant policy access to).

Step 2.14 — Create a Kubernetes auth role

This is the binding: which ServiceAccount, in which namespace, gets which policy.

kubectl create serviceaccount myapp-sa -n default

kubectl exec -n vault -it vault-0 -- vault write auth/kubernetes/role/myapp-role \
  bound_service_account_names=myapp-sa \
  bound_service_account_namespaces=default \
  policies=myapp-read \
  ttl=1h

Step 2.15 — Prove the trust chain manually

This is the step worth not skipping — before Injector or VSO ever wraps this in automation, do it by hand once so the mechanics are real to you.

One HA-specific thing to notice as you do this: the vault Kubernetes Service load-balances across all three pods, not just the active one. If your request happens to land on vault-1 or vault-2 (a standby), Vault transparently forwards it to the active node and returns the result — this is core HA request forwarding, not something you configured. You don’t need to target vault-0 specifically for reads/writes through the Service; you only had to target it directly in Steps 2.5–2.7 because init/unseal/raft join are node-specific operations by nature.

Spin up a throwaway pod using that ServiceAccount:

kubectl run vault-test -n default \
  --image=curlimages/curl \
  --overrides='{"apiVersion":"v1","kind":"Pod","spec":{"serviceAccountName":"myapp-sa"}}' \
  --command -- sleep 3600

Exec into it:

kubectl exec -it vault-test -n default -- sh

Inside that shell, read the pod’s own ServiceAccount token and use it to log in to Vault:

KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

curl --request POST \
  --data "{\"jwt\": \"$KUBE_TOKEN\", \"role\": \"myapp-role\"}" \
  http://vault.vault.svc.cluster.local:8200/v1/auth/kubernetes/login

Expected result: a JSON response containing auth.client_token — a Vault token, scoped to the myapp-read policy, issued purely because this pod’s identity matched the role you configured. No credentials were typed anywhere.

Take that token and read the secret with it:

VAULT_TOKEN="<client_token from above>"

curl --header "X-Vault-Token: $VAULT_TOKEN" \
  http://vault.vault.svc.cluster.local:8200/v1/secret/data/myapp/config

Expected: JSON containing username: demo, password: demo123.

Exit the pod and clean it up:

exit
kubectl delete pod vault-test -n default

Step 2.16 — Enable the Vault UI (optional)

helm upgrade vault hashicorp/vault \
  --namespace vault \
  --reuse-values \
  --set 'ui.enabled=true' \
  --set 'server.affinity=' \
  --server-side=false

Two things about this command worth understanding rather than just running:

• --server-side=false avoids the MutatingWebhookConfiguration caBundle ownership conflict covered in the troubleshooting table below — Helm’s server-side apply collides with the injector controller’s own management of that field on any upgrade to this release.
• --set 'server.affinity=' is included again here, deliberately. If you fixed stuck Pending pods earlier via a direct kubectl patch statefulset (rather than a successful helm upgrade), that fix only lives on the live StatefulSet object — it was never recorded in Helm’s stored release values. Any later helm upgrade --reuse-values regenerates the manifest from those stale values and will silently reintroduce the anti-affinity rule, undoing your earlier fix. Keep this flag on every helm upgrade for this release until you’ve confirmed it’s actually stored (helm get values vault -n vault should show it).

ui.enabled is baked into the server config, which is only read at container startup, so recreate all three pods:

kubectl delete pod vault-0 vault-1 vault-2 -n vault
kubectl get pods -n vault -w

Once all three show 0/1 Running (sealed, not Pending), unseal and re-login vault-0, then re-check vault-1/vault-2 (raft list-peers — they generally don’t need to rejoin, just unseal again).

Reach the UI from your Mac via port-forward — minikube’s podman driver doesn’t give clean direct access to the node IP from the host:

kubectl port-forward -n vault svc/vault 8200:8200

Leave that running and open http://127.0.0.1:8200/ui in a browser. Log in with your root token. The port-forward only lives as long as that terminal stays open.

Everything the Agent Injector and Vault Secrets Operator do in Phase 3 is this exact exchange, automated. Injector does it via a sidecar that intercepts pod startup; VSO does it via a controller watching CRDs and syncing the result into a Kubernetes Secret. Different delivery mechanisms, identical trust chain underneath — which is exactly the “which one should I use” distinction that comes up in customer conversations.

Phase 2 troubleshooting

Fixing stuck anti-affinity Pending pods, in full

If you’re following Step 2.3 as written above (with --set 'server.affinity=' included from the start), you shouldn’t hit this at all — it only bites if you installed before that line was added, or changed server.ha.replicas after the fact. If you do hit it:

1. Remove the anti-affinity rule from the StatefulSet’s pod template:

kubectl patch statefulset vault -n vault \
  -p '{"spec":{"template":{"spec":{"affinity":null}}}}'

2. The subtlety: this only changes the template for pods created from now on. vault-0 was already created under the old template, and pod anti-affinity is symmetric — vault-0’s own still-active rule refuses to let any matching pod land on its node, which blocks vault-1/vault-2 from scheduling even after the patch. You have to recreate vault-0 too, not just the two stuck pods:

kubectl delete pod vault-0 vault-1 vault-2 -n vault
kubectl get pods -n vault -w

vault-0’s data is safe (it reattaches to its existing PVC via Raft storage), but its seal state is in-memory only — it comes back sealed, and its ~/.vault-token session (on an emptyDir volume) is wiped. You’ll need to re-unseal and re-login before continuing:

kubectl exec -n vault -it vault-0 -- vault operator unseal <original-unseal-key>
kubectl exec -n vault -it vault-0 -- vault login <original-root-token>

Once vault-0 is back and unsealed, vault-1/vault-2 should clear Pending on their own — then resume the raft-join/unseal steps (2.7) for both.

Assumes Phase 2 is done and working: a 3-node unsealed HA cluster running in vault namespace, kubernetes auth configured, myapp-role bound to myapp-sa/default, policy myapp-read scoped to secret/data/myapp/*, test secret at secret/myapp/config. Any vault exec commands in this phase assume you’re still logged in from Step 2.8 (root token session on vault-0) — if that pod restarted, re-run vault login <root-token> first.

Three separate pathways, three separate mechanisms. You’ll stand each one up and prove it independently.

Phase 3A: Vault Agent Injector

Step 3A.1 — Confirm it’s already running

The hashicorp/vault Helm chart installs the Injector by default alongside the server — you didn’t need to opt in. Confirm:

kubectl get pods -n vault -l app.kubernetes.io/name=vault-agent-injector

Expected: one pod, Running, 1/1. If it’s not there, the chart was likely installed with injector.enabled=false somewhere — check with helm get values vault -n vault.

Step 3A.2 — Deploy a test workload with injection annotations

The Injector works via a mutating webhook: annotations on a pod spec tell it to add a Vault Agent sidecar that authenticates, fetches the secret, and writes it to a shared volume as a file.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: injector-demo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: injector-demo
  template:
    metadata:
      labels:
        app: injector-demo
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "myapp-role"
        vault.hashicorp.com/agent-inject-secret-config.txt: "secret/data/myapp/config"
    spec:
      serviceAccountName: myapp-sa
      containers:
      - name: app
        image: curlimages/curl
        command: ["sleep", "3600"]
EOF

Note what’s doing the work: the annotations, not a new controller you installed. vault.hashicorp.com/role maps directly to the myapp-role you created in Step 2.14. Nothing about auth changed — same ServiceAccount, same role, same policy.

Step 3A.3 — Verify the sidecar and the rendered secret

kubectl get pods -n default -l app=injector-demo

Expected: 2/2 containers ready — your app container plus the injected vault-agent sidecar. If you’re still at 1/1 after a minute with no errors, see the Phase 3A troubleshooting table below — that’s a specific, silent failure mode, not a timing issue.

kubectl exec -n default deploy/injector-demo -c app -- cat /vault/secrets/config.txt

Expected: the rendered secret data (username: demo, password: demo123 by default template format). The Agent wrote this file before your app container’s process even started — that’s the ordering guarantee the Injector gives you.

Clean up when done:

kubectl delete deployment injector-demo -n default

Phase 3A troubleshooting

Phase 3B: Vault Secrets Operator (VSO)

Different model: instead of a sidecar rendering a file at pod startup, a cluster-wide controller watches CRDs and continuously syncs Vault data into native Kubernetes Secret objects — which any workload can then mount or reference normally, with no annotations or sidecar required.

Step 3B.1 — Install VSO

helm install vault-secrets-operator hashicorp/vault-secrets-operator \
  --namespace vault-secrets-operator-system \
  --create-namespace

Verify:

kubectl get pods -n vault-secrets-operator-system

Expected: the controller pod Running.

Step 3B.2 — Create a VaultConnection

Tells VSO where Vault lives.

cat <<EOF | kubectl apply -f -
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: vault-connection
  namespace: default
spec:
  address: http://vault.vault.svc.cluster.local:8200
EOF

Step 3B.3 — Create a VaultAuth

Tells VSO how to authenticate — reusing the same Kubernetes auth mount and role from Phase 2.

cat <<EOF | kubectl apply -f -
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
  namespace: default
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: myapp-role
    serviceAccount: myapp-sa
  vaultConnectionRef: vault-connection
EOF

Same role, same ServiceAccount, same trust chain you proved by hand in Step 2.15 — VSO is just automating that curl exchange on a schedule.

Step 3B.4 — Create a VaultStaticSecret

This is the actual sync directive: read this Vault path, write it into this Kubernetes Secret, refresh on this interval.

cat <<EOF | kubectl apply -f -
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: myapp-config
  namespace: default
spec:
  vaultAuthRef: vault-auth
  mount: secret
  type: kv-v2
  path: myapp/config
  refreshAfter: 30s
  destination:
    name: myapp-config-secret
    create: true
EOF

Step 3B.5 — Verify the synced Secret

kubectl get secret myapp-config-secret -n default -o jsonpath='{.data.username}' | base64 -d
echo
kubectl get secret myapp-config-secret -n default -o jsonpath='{.data.password}' | base64 -d

Expected: demo and demo123 — same values as the Injector pulled, but this time they landed in an ordinary Kubernetes Secret with no sidecar involved. Any pod can now reference myapp-config-secret with a standard envFrom/volumeMount, same as any other K8s Secret.

Prove it’s live-syncing: change the Vault value and watch it propagate without touching Kubernetes at all.

kubectl exec -n vault -it vault-0 -- vault kv put secret/myapp/config username=demo password=updated456

Wait ~30s (your refreshAfter), then re-check:

kubectl get secret myapp-config-secret -n default -o jsonpath='{.data.password}' | base64 -d

Expected: updated456. This is the core difference from Injector — VSO keeps syncing on an interval; Injector renders once at pod startup (or on a template-configured interval of its own, but it’s sidecar-local, not a cluster-wide Secret object).

Clean up when done:

kubectl delete vaultstaticsecret myapp-config -n default
kubectl delete secret myapp-config-secret -n default

Phase 3C: cert-manager (with Vault PKI as the issuer)

This is the pattern that matters for customer conversations: cert-manager handling the Kubernetes-side automation (Certificate CRDs, renewal, Secret lifecycle) while Vault’s PKI secrets engine is the actual certificate authority signing things.

Step 3C.1 — Install cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true

Verify:

kubectl get pods -n cert-manager

Expected: three pods running (cert-manager, cert-manager-cainjector, cert-manager-webhook).

Step 3C.2 — Enable and configure Vault’s PKI secrets engine

Back to vault-0:

kubectl exec -n vault -it vault-0 -- sh -c '
vault secrets enable pki
vault secrets tune -max-lease-ttl=8760h pki
vault write -field=certificate pki/root/generate/internal \
  common_name="lab.local" ttl=8760h > /dev/null
vault write pki/config/urls \
  issuing_certificates="http://vault.vault.svc.cluster.local:8200/v1/pki/ca" \
  crl_distribution_points="http://vault.vault.svc.cluster.local:8200/v1/pki/crl"
vault write pki/roles/lab-dot-local \
  allowed_domains="lab.local" \
  allow_subdomains=true \
  max_ttl="72h"
'

This generates a self-contained root CA inside Vault (fine for a lab; a real deployment would use an intermediate CA signed by something external) and defines a role that constrains what domains it’ll issue certs for.

Step 3C.3 — Give cert-manager a way to authenticate to Vault

For lab simplicity, use a scoped Vault token rather than wiring up a second Kubernetes-auth role (production deployments typically use serviceAccountRef-based Kubernetes auth here instead — worth doing later, but it adds RBAC/TokenRequest plumbing that isn’t the point of this exercise).

kubectl exec -n vault -it vault-0 -- sh -c '
vault policy write cert-manager-pki - <<EOF
path "pki/sign/lab-dot-local" {
  capabilities = ["create", "update"]
}
EOF
vault token create -policy=cert-manager-pki -period=768h -field=token
'

Copy the token that’s printed, then store it as a Kubernetes Secret:

kubectl create secret generic cert-manager-vault-token \
  -n default \
  --from-literal=token=<paste-token-here>

Step 3C.4 — Create the cert-manager Issuer

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: default
spec:
  vault:
    path: pki/sign/lab-dot-local
    server: http://vault.vault.svc.cluster.local:8200
    auth:
      tokenSecretRef:
        name: cert-manager-vault-token
        key: token
EOF

Verify it’s healthy:

kubectl describe issuer vault-issuer -n default

Look for Ready: True in the conditions.

Step 3C.5 — Request a certificate

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myapp-cert
  namespace: default
spec:
  secretName: myapp-tls
  duration: 24h
  renewBefore: 12h
  commonName: myapp.lab.local
  dnsNames:
  - myapp.lab.local
  issuerRef:
    name: vault-issuer
    kind: Issuer
EOF

Step 3C.6 — Verify issuance

kubectl get certificate myapp-cert -n default

Expected: READY becomes True within a few seconds.

kubectl get secret myapp-tls -n default -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout | grep -A1 Issuer

Expected: the issuer CN traces back to lab.local — the root CA you generated in Step 3C.2.

Step 3C.7 — Consume the cert in a test workload

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: cert-consumer
  namespace: default
spec:
  containers:
  - name: app
    image: curlimages/curl
    command: ["sleep", "3600"]
    volumeMounts:
    - name: tls
      mountPath: /etc/certs
      readOnly: true
  volumes:
  - name: tls
    secret:
      secretName: myapp-tls
EOF

kubectl exec -n default cert-consumer -- ls /etc/certs
kubectl exec -n default cert-consumer -- sh -c "cat /etc/certs/tls.crt | head -1"

Expected: ca.crt, tls.crt, tls.key present, and the first line of tls.crt is -----BEGIN CERTIFICATE-----.

Prove renewal works: cert-manager watches renewBefore and rotates automatically as expiry approaches. You don’t need to wait 12 hours to see the mechanism — the point to understand is that myapp-tls is a live-managed Secret, not a one-time artifact; anything mounting it will pick up rotated certs on the next volume refresh without you touching the Certificate resource again.

Clean up when done:

kubectl delete pod cert-consumer -n default
kubectl delete certificate myapp-cert -n default
kubectl delete secret myapp-tls -n default

Phase 3 wrap-up: the three pathways compared

This is the exact framing customers ask for — “which one do I use” isn’t a maturity question, it’s a delivery-model question. You now have working, hand-built proof of all three, not just the marketing description of them.

Any time a Vault pod restarts (a helm upgrade, a manual kubectl delete pod, a node reboot), you’ll need some subset of these. Bookmark this section — you’ll come back to it more than any single numbered step.

Check cluster/pod state:

kubectl get pods -n vault
kubectl exec -n vault -it vault-0 -- vault status

Unseal a pod (needed any time a pod restarts — seal state is in-memory only):

kubectl exec -n vault -it <pod-name> -- vault operator unseal <unseal-key>

Re-authenticate your CLI session (needed any time a pod restarts — the token lives on an emptyDir volume that doesn’t survive):

kubectl exec -n vault -it vault-0 -- vault login <root-token>

Check Raft peer health:

kubectl exec -n vault -it vault-0 -- vault operator raft list-peers

Reach the UI:

kubectl port-forward -n vault svc/vault 8200:8200
# then open http://127.0.0.1:8200/ui in a browser

Any helm upgrade on this release — always include these two flags regardless of what you’re changing, to avoid the two recurring gotchas covered in Phase 2’s troubleshooting table:

helm upgrade vault hashicorp/vault \
  --namespace vault \
  --reuse-values \
  --set 'server.affinity=' \
  --server-side=false \
  --set 'whatever.you.are.actually.changing=value'

If a pod restart breaks scheduling again (stuck Pending, “does not have a host assigned”): see Phase 2’s “Fixing stuck anti-affinity Pending pods, in full” — the short version is you likely need to recreate all three pods, not just the stuck ones.