Unit 12 — Labs

Chart and version mapping — from helm search repo hashicorp/vault -l:

Supported Kubernetes minor releases for both the Vault Helm chart and VSO: 1.36, 1.35, 1.34, 1.33, 1.32. IBM tests and verifies Vault against these; IBM does not support Vault with other versions of Kubernetes, although the functionality may work. Check your Minikube version against that list before troubleshooting anything else.

The chart is not compatible with Helm 2. Use Helm 3.6+. VSO requires Helm 3.7+ and a Kubernetes cluster running 1.23+.

Always run Helm with --dry-run before any install or upgrade to verify changes. The docs repeat this warning twice; treat it as procedure, not advice.

Two warnings that shape the whole lab:

• By default, the chart runs in standalone mode. This mode uses a single Vault server with a file storage backend. This is a less secure and less resilient installation that is NOT appropriate for a production setup.
• While the Helm chart automatically sets up complex resources and exposes the configuration to meet your requirements, it does not automatically operate Vault. You are still responsible for learning how to monitor, backup, upgrade, etc. the Vault cluster.

A.1 Add the repo and inspect available versions

helm repo add hashicorp https://helm.releases.hashicorp.com
helm search repo hashicorp/vault
helm search repo hashicorp/vault -l

A.2 Install

Pin the version — Helm will install the latest chart found in a repo by default. It’s recommended to specify the chart version.

helm install vault hashicorp/vault --version 0.34.0

Overriding values, both forms:

helm install vault hashicorp/vault --set "server.dev.enabled=true"

cat override-values.yml
server:
  ha:
    enabled: true
    replicas: 5

helm install vault hashicorp/vault --values override-values.yml

⚠ HA mode caveat — read before using server.ha.enabled=true. The Helm chart’s HA mode documentation states it installs three Vault servers with an existing Consul storage backend, and suggests installing Consul via the Consul Helm chart. Your lab is described as integrated storage (Raft), not Consul.

The docs point to an enterprise-with-raft examples page for Enterprise deployments, and note that for Vault Enterprise deployments on Kubernetes 1.35+, you can enable redundancy zones for improved availability and fault tolerance across availability zones.

If your lab already runs Raft HA, keep your working values file. Do not adopt the server.ha.enabled snippet above expecting Raft — consult the chart’s enterprise-with-raft example and configuration reference for the correct values.

A.3 Initialize and unseal

kubectl get pods -l app.kubernetes.io/name=vault

kubectl exec -ti vault-0 -- vault operator init

The output displays the key shares and initial root key generated.

kubectl exec -ti vault-0 -- vault operator unseal   # Unseal Key 1
kubectl exec -ti vault-0 -- vault operator unseal   # Unseal Key 2
kubectl exec -ti vault-0 -- vault operator unseal   # Unseal Key 3

Repeat the unseal process for all Vault server pods. When all Vault server pods are unsealed they report READY 1/1.

A.4 Access the UI

The Vault UI is enabled but NOT exposed as service for security reasons.

kubectl port-forward vault-0 8200:8200

A.5 Learn the leader-status labels

Vault has K8s service discovery built in (when enabled in the server configuration) and will automatically change the labels of the pod with its current leader status.

kubectl get pods -l vault-active=false   # standbys
kubectl get pods -l vault-active=true    # active

Learn these now — Lab D depends on them.

A.6 Characterize the deployment

This is the drill to run against any customer’s Vault.

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='<root-token>'

vault status
vault operator raft list-peers
vault operator raft autopilot state
vault secrets list -detailed
vault auth list -detailed
vault policy list
vault audit list -detailed

Read every field of vault status — seal type, storage type, HA mode, version. Map what you see onto Unit 2’s reference architecture and note every deviation.

Purpose: make Unit 3’s “Vault stops serving requests” property concrete. Do this before anything else, because it’s the property that surprises people.

B.1 Enable two devices of different types

vault audit enable file file_path=/vault/vault-audit.log
vault audit enable syslog tag="vault" facility="AUTH"

The Vault process user must have write access to the location of the file. If it doesn’t, you get:

* sanity check failed; unable to open "/vault/vault-audit.log" for writing: open /vault/vault-audit.log: permission denied

Vault Helm includes a configurable auditStorage option that provisions a persistent volume to store audit logs. Consult the chart’s configuration reference for the exact value syntax before deciding where the file device writes.

B.2 Verify hashing and the audit-hash API

Most strings contained within requests and responses are hashed with a salt using HMAC-SHA256. You are still able to check the value of secrets by generating HMACs yourself.

vault write /sys/audit-hash/file input="invalid username or password"

Then grep the audit log for that hash. This is the exact technique from the incident-response procedure in Unit 10.5 — practice it before you need it.

B.3 Observe a blocking failure

Do this only in the lab. The mechanism is documented; the specific exercise is mine.

Fill the audit volume, or make the file path unwritable, and watch Vault stop serving requests. Then confirm the metric moves:

vault.audit.log_request_failure    # alert threshold: > 0
vault.audit.log_response_failure   # alert threshold: > 0

When Vault cannot log to any of the configured audit log devices it ceases all user operations. Having watched it once, you’ll never again describe audit devices as optional in a customer design review.

Follow Unit 3.4 in order. Steps 1–2 are Lab A and B. The remainder:

C.1 Operator policy in the root namespace

export VAULT_TOKEN=<your root token>
vault policy write operator operator-policy.hcl

Policy content is in Unit 3.2.

C.2 Namespaces

Namespace creation syntax, from the RACF validated pattern:

vault namespace create admin
vault namespace create -namespace=admin z

Substitute your own tenant name for z. Deploy tenant namespaces under the admin namespace, per the structure in Unit 4.2.

C.3 Tenant admin policy

export VAULT_NAMESPACE=admin
vault policy write admin admin-policy.hcl

Policy content is in Unit 3.2.

C.4 First auth method

Use LDAP or OIDC per Unit 5.5 if you have an IdP reachable from the lab. If not, the RACF pattern’s userpass approach works for exercising the sequence — with the pattern’s own warning that this is for instructional purposes and production must use a more secure method.

C.5 Revoke the root token

vault login <your root token>
vault token revoke <your root token>

Do this. The point of the lab is to experience operating Vault without a root token, and to practice vault operator generate-root when you inevitably need one back.

This is the highest-value lab in the set, because the sequence is non-obvious and getting it wrong in production causes downtime.

D.1 The rules

The Vault StatefulSet uses OnDelete update strategy. It is critical to use OnDelete instead of RollingUpdate because standbys must be updated before the active primary. A failover to an older version of Vault must always be avoided.

Always back up your data before upgrading. Vault does not make backward-compatibility guarantees for its data store. Simply replacing the newly-installed Vault binary with the previous version may not cleanly downgrade Vault, as upgrades may perform changes to the underlying data structure that make the data incompatible with a downgrade.

D.2 Dry run first

helm upgrade vault hashicorp/vault --version=0.34.0 \
    --set='server.image.repository=vault' \
    --set='server.image.tag=123.456' \
    --dry-run

This should cause no changes (although the resources are updated). If everything is stable, helm upgrade can be run.

D.3 The pod replacement sequence

The helm upgrade command should have updated the StatefulSet template, however, no pods have been deleted. The pods must be manually deleted to upgrade. Deleting the pods does not delete any persisted data.

For HA, you must upgrade your standby pods before upgrading the active pod:

1. Before deleting the standby pod, remove the associated node from the raft with vault operator raft remove-peer <server_id>.
2. Confirm Vault removed the node successfully from Raft with vault operator raft list-peers.
3. Once you confirm the removal, delete the pod.

Why the remove-peer step exists: removing a pod without first deleting the node from its cluster means that Raft will not be aware of the correct number of nodes in the cluster. Not knowing the correct number of nodes can trigger a leader election, which can potentially cause unneeded downtime.

Then:

kubectl get pods -l vault-active=false
kubectl delete pod <name of Vault pod>

Sequentially delete every pod that is not the active primary, ensuring the quorum is maintained at all times.

If auto-unseal is not being used, the newly scheduled Vault standby pods need to be unsealed:

kubectl exec -ti <name of pod> -- vault operator unseal

Finally, once the standby nodes have been updated and unsealed, delete the active primary, and unseal it too. After a few moments the Vault cluster should elect a new active primary.

D.4 Deliberately do it wrong

Skip the remove-peer step on one standby and watch for the leader election. Observing this once is the best way to internalize why the sequence matters.

E.1 Automated snapshots

vault write sys/storage/raft/snapshot-auto/config/[:name] [options]

Consult /vault/api-docs/system/storage/raftautosnapshots for the available options.

E.2 Restore to a fresh cluster

Per Unit 10.6: initialize a new cluster, then:

vault operator raft snapshot restore -force <backup.snap>
vault operator unseal <unseal_key>

The original unseal keys will be needed following restore — the new cluster’s initialization keys are temporary.

E.3 The exercise that matters

Restore a snapshot into a cluster and confirm you must use the original cluster’s unseal keys, not the new cluster’s. Since your lab is Shamir, this works — and it demonstrates why you can safely use -force when restoring a snapshot from a system using Shamir seal.

Then reason through (don’t attempt) the auto-unseal case: with autounseal, it is crucial to use the same seal configuration for the new cluster as the one used for the cluster being restored. If not, data decryption will not be possible after the restore, and using -force will disable a valuable safeguard.

The configuration commands in Units 7 and 8 run as written. The sequence below orders them into a working progression.

F.1 Database secrets engine

Deploy a PostgreSQL into the cluster by whatever means your environment allows, then work through Unit 7.5 in order:

1. Configure the connection with allowed_roles set explicitly.
2. vault write -force <mount>/rotate-root/<name> and confirm the original password no longer works.
3. Create a dynamic role from the HVD readonly.sql creation statement.
4. vault read <mount>/creds/<role> and verify the credential works and is correctly scoped — test the deny path, not just the allow path.
5. Walk the lease commands: vault list sys/leases/lookup/..., vault lease lookup, vault lease revoke.
6. Create a static role with rotation_statements and observe that repeated reads return the same password until rotation.

An exercise worth adding: apply username customization by prefixing generated roles with vault_managed, then confirm it appears in \du output. Unit 7.5 explains why — it can help prevent accidental deletion of Vault’s managed accounts.

F.2 PKI

Work through Unit 8. The role definition in 8.7 is HVD’s and runs as written.

Test every deny path: wildcard CN, the tenant-1xyz.example.com prefix attack, and an out-of-tenant domain. Then request a TTL longer than max_ttl and confirm it is silently capped rather than rejected — requests for certificates with longer TTLs are overridden and issued with the maximum allowed value.

Building the external root CA. HVD requires the root CA to live outside Vault but does not publish OpenSSL commands for creating one. Use your organization’s existing CA, or construct the root and sign the Vault intermediate with your own tooling.

For name constraints, HVD states the signing CA can impose URI and DNS domain constraints, exclusions, and path length controls on tenant issuers. Confirm the exact parameter names against the PKI API reference before building the tenant-delegation demonstration.

F.3 Templated policy and the 2.0.1 restriction

Set entity metadata to a literal value, confirm a templated policy works, then set it to a value containing a glob and confirm the request is denied. The behavior is documented (Unit 5.4); the exercise is mine. Worth doing once so you recognize it instantly during a customer upgrade.

G.1 Kubernetes auth method

vault auth enable kubernetes

vault write auth/kubernetes/config \
  token_reviewer_jwt="<your reviewer service account JWT>" \
  kubernetes_host=https://192.168.99.100:<port> \
  kubernetes_ca_cert=@ca.crt

vault write auth/kubernetes/role/demo \
  bound_service_account_names=myapp \
  bound_service_account_namespaces=default \
  policies=default \
  audience=myapp \
  ttl=1h

Use kubectl cluster-info to validate the Kubernetes host address and TCP port.

The alias_name_source and bound_service_account_namespace_selector behaviors are in Unit 9.5. Test a negative case — a service account not bound to the role — and confirm login fails.

G.2 Install VSO

helm repo add hashicorp https://helm.releases.hashicorp.com
helm search repo hashicorp/vault-secrets-operator

helm install --version 1.5.0 --create-namespace \
  --namespace vault-secrets-operator \
  vault-secrets-operator hashicorp/vault-secrets-operator

As of VSO 0.8.0, VSO will automatically update its CRDs. No manual CRD step is needed.

Kustomize is the alternative, but Kustomize does not support all features of the Helm chart. Notably it will not deploy default VaultAuthMethod, VaultConnection or Transit related resources. Kustomize also does not support pre-delete hooks that the Helm chart uses to cleanup resources and remove finalizers on the uninstall path. Use Helm.

Verify:

kubectl get pods -n vault-secrets-operator

G.3 The CRD model

VaultConnection and VaultAuth CRDs provide Vault connection and authentication configuration information for the operator. Consider VaultConnection and VaultAuth as foundational resources used by all secret replication type resources. Then one secret CRD per secret you want synced.

Supported Vault versions: Vault Enterprise/Community 1.11+, HCP Vault Dedicated 1.11+.

Supported auth methods: Kubernetes (relies on short-lived Kubernetes ServiceAccount tokens), JWT (either static JWT tokens or short-lived Kubernetes ServiceAccount tokens), AppRole (relies on static AppRole credentials), AWS, GCP.

Note how that list maps onto Unit 6: platform-native first, AppRole as the static-credential fallback.

VaultConnection:

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: vso-example
  name: vault-connection
spec:
  # required configuration
  # address to the Vault server.
  address: http://vault.vault.svc.cluster.local:8200

  # optional configuration
  # HTTP headers to be included in all Vault requests.
  # headers: []
  # TLS server name to use as the SNI host for TLS connections.
  # tlsServerName: ""
  # skip TLS verification for TLS connections to Vault.
  # skipTLSVerify: false
  # the trusted PEM encoded CA certificate chain stored in a Kubernetes Secret
  # caCertSecretRef: ""

VaultAuth:

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  namespace: vso-example
  name: vault-auth
spec:
  # required configuration
  # VaultConnectionRef of the corresponding VaultConnection CustomResource.
  # If no value is specified the Operator will default to the `default` VaultConnection,
  # configured in its own Kubernetes namespace.
  vaultConnectionRef: vault-connection
  # Method to use when authenticating to Vault.
  method: kubernetes
  # Mount to use when authenticating to auth method.
  mount: kubernetes
  # Kubernetes specific auth configuration, requires that the Method be set to kubernetes.
  kubernetes:
    # role to use when authenticating to Vault
    role: example
    # ServiceAccount to use when authenticating to Vault
    # it is recommended to always provide a unique serviceAccount per Pod/application
    serviceAccount: default

  # optional configuration
  # Vault namespace where the auth backend is mounted (requires Vault Enterprise)
  # namespace: ""
  # Params to use when authenticating to Vault
  # params: []
  # HTTP headers to be included in all Vault authentication requests.
  # headers: []

Note “it is recommended to always provide a unique serviceAccount per Pod/application” — the same least-privilege principle as one-Agent-per-application in Unit 8.10.

Note also the namespace field: Vault namespace where the auth backend is mounted (requires Vault Enterprise). With the Unit 4.2 hierarchy, this is where your tenant namespace goes. VSO also supports cross Vault namespace authentication for Vault Enterprise 1.13+.

VaultAuthGlobal (VSO v0.8.0+) is optional: a namespaced resource that provides shared Vault authentication configuration that can be inherited by multiple VaultAuth custom resources. It reduces config duplication but cannot be used for authentication directly.

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuthGlobal
metadata:
  namespace: vso-example
  name: vault-auth-global
spec:
  defaultAuthMethod: kubernetes
  kubernetes:
    audiences:
    - vault
    mount: kubernetes
    namespace: example-ns
    role: auth-role
    serviceAccount: default
    tokenExpirationSeconds: 600
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  namespace: vso-example
  name: vault-auth
spec:
  vaultAuthGlobalRef:
    name: vault-auth-global
  kubernetes:
    role: local-role

The kubernetes.role field in the VaultAuth custom resource spec overrides the value of the corresponding field in the VaultAuthGlobal custom resource. All other fields are inherited.

G.4 Sync a static secret

Supported secrets engines: kv-v2, kv-v1.

KV v2 — results in requests to http://127.0.0.1:8200/v1/kvv2/eng/apikey/google:

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: vso-example
  name: vault-static-secret-v2
spec:
  vaultAuthRef: vault-auth
  mount: kvv2
  type: kv-v2
  path: eng/apikey/google
  version: 2
  refreshAfter: 60s
  destination:
    create: true
    name: static-secret2

KV v1 differs only in type: kv-v1 and no version field.

Verify the destination Kubernetes Secret appears, change the value in Vault, and watch it re-sync after refreshAfter.

G.5 Sync a dynamic database credential

Connect this to Lab F.1 — point mount at the database engine you configured there.

To send requests to http://127.0.0.1:8200/v1/db/creds/my-postgresql-role to generate a new credential:

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  namespace: vso-example
  name: vault-dynamic-secret-db
spec:
  vaultAuthRef: vault-auth
  mount: db
  path: creds/my-postgresql-role
  destination:
    create: true
    name: dynamic-db

the exercise: watch the lease in vault list sys/leases/lookup/... while VSO manages it, then delete the VSO resource and confirm the lease behavior. This is the Unit 7 lease model observed through the operator.

G.6 Sync a PKI certificate

Connect this to Lab F.2. Supported secrets engine: pki. Results in a request to http://127.0.0.1:8200/v1/pki/issue/default:

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultPKISecret
metadata:
  namespace: vso-example
  name: vault-pki
spec:
  vaultAuthRef: vault-auth
  mount: pki
  role: default
  commonName: example.com
  format: pem
  expiryOffset: 1s
  ttl: 60s
  namespace: tenant-1
  destination:
    create: true
    name: pki1

Note this uses the issue endpoint — consistent with Unit 8.9, where issue is required for automated workflows. expiryOffset is VSO’s analogue of Vault Agent’s lease_renewal_threshold. Set a short ttl and watch re-issuance.

G.7 Rollout restart

VSO supports performing a rollout-restart upon secret rotation or during drift remediation, and automatic secret rotation for Deployment, ReplicaSet, StatefulSet Kubernetes resource types.

This is VSO’s answer to the Unit 8.10 problem — NGINX not reloading when a certificate changes on disk. Where Vault Agent uses a post-render exec, VSO restarts the workload.

deploy a workload consuming a synced secret, force a rotation, and confirm the restart happens. Compare the two approaches side by side; that comparison is the substance of the Unit 9.4 decision.

G.8 Client cache and instant updates

VSO can optionally cache Vault client information such as Vault tokens and leases in Kubernetes Secrets within its own namespace. The client cache enables seamless upgrades because Vault tokens and dynamic secret leases can continue to be tracked and renewed through leadership changes. Client cache persistence and encryption is not enabled by default because it requires extra configuration and Vault Server setup. VSO supports encrypting the client cache using Vault Server’s transit secrets engine.

Read that against Unit 10.7: leases are cluster-local under performance replication, and VSO’s cache is what keeps them tracked across VSO’s own restarts.

Instant updates (Enterprise, VSO v0.8.0+): VSO can instantly update Kubernetes Secrets when changes are made in Vault, by subscribing to Vault Events for change notification. Requires Vault Enterprise 1.16.3+, and supports VaultStaticSecret only (kv-v1, kv-v2).

Setting a refresh interval is still recommended since event message delivery is not guaranteed. Do not remove refreshAfter when you enable instant updates — that’s the trap.

Consult the VSO documentation for client-cache and instant-updates enabling configuration.

H.1 Telemetry stanza

telemetry {
  disable_hostname = true
  prometheus_retention_time = "12h"
}

• prometheus_retention_time (string: “24h”): specifies the amount of time that Prometheus retains metrics in memory. Setting this to 0 disables Prometheus telemetry.
• disable_hostname (boolean): prevents the system from prefixing metrics with hostname.

Why disable_hostname = true matters: by default, the system scrapes metrics from the leader node, which may change over the lifetime of a cluster. If disable_hostname is not set to true, the system prefixes metrics with the hostname of the leader node, which causes the system to lose metrics when the leader changes.

That is a real gotcha — a customer whose metrics vanish after a leader election almost certainly has this unset.

If you do not already have an existing metrics platform, HashiCorp recommends Prometheus, which has a good balance of simplicity of configuration and scalability.

H.2 Health check endpoint

The HVD load balancer health check path:

/v1/sys/health?perfstandbyok=true&activecode=200&performancestandbycode=473

And the simpler documented form: configure your load balancer to perform a health check to all Vault nodes using the path /v1/sys/health?perfstandbyok=true, with an expected status code of 200.

By default, only a leader node returns a 200 status code. Setting the perfstandbyok parameter to true instead directs all nodes to return a 200 status code, including non-leader nodes. Configuring the health check in this way enables the load balancer to spread client read requests to all nodes in the cluster, while the system still redirects write requests sent to performance standby nodes to the leader node using request forwarding.

This is the load balancer configuration that Unit 8.5 says is required for no_store=true PKI throughput gains to materialize. Connect those two facts — it’s the answer to “we enabled no_store and saw no improvement.”

The Helm chart also exposes probes:

server:
  readinessProbe:
    enabled: true
    path: '/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204'

Using this customized probe, a postStart script could automatically run once the pod is ready for additional setup.

H.3 Transactional monitoring

Run the script from Unit 10.4 on a schedule against your lab. Measure success/failure and execution time at each step.

H.4 Watch metrics move

Drive load through the lab and correlate against the thresholds in Unit 10.2. Specifically pair vault.core.handle_login_request with vault.token.creation — Unit 10.2 explains why a spike in the first without a rise in the second is the signal to investigate immediately.

• Production-grade TLS. The Helm examples use tls_disable = 1 in the HA config snippets. Unit 2.4 requires TLS passthrough with certificates distributed to the nodes; see the chart’s standalone-tls example.
• Auto-unseal. The chart supports Google KMS and AWS KMS auto-unseal with documented config examples, but exercising it needs cloud credentials your lab may not have. The seal stanza forms for AWS KMS, Azure Key Vault, GCP Cloud KMS, and PKCS11 are all in the HVD Detailed design page if you want to read them.
• Replication. Needs a second cluster.
• Sentinel. The example policies in Units 5 and 8 are HVD’s; consult the Sentinel documentation for sys/policies/egp write syntax and enforcement level semantics before deploying them.