Unit 6B — Static Secrets and the KV Engine

A static secret is any sensitive credential, such as a password or API key, that you create and maintain over a long period of time. The secret value remains the same until you change or rotate it.

Securing static secrets protects your organization from data breaches and unauthorized access to critical systems. Organizations classify these credentials as restricted because compromised secrets can cause severe damage, including data loss, security breaches, and compliance violations.

Vault’s key/value (KV) secrets engine allows you to store any type of arbitrary secret such as usernames, passwords, API keys, and even static TLS certificates that are not able to be automatically provisioned by base64 encoding the certificate.

You can protect and manage access to the secrets stored in Vault’s KV secrets engine using Vault’s robust access control policy framework, and incorporate advanced Sentinel policies to ensure only authorized individuals or applications access the secrets they are entitled to.

Note the framing tension worth naming with customers: while you often classify secrets as restricted, you still need to use these secrets to access systems or configure applications. Storage alone isn’t the goal — controlled access is.

The KV secrets engine is a generic key-value store used to store arbitrary secrets within the configured physical storage for Vault. Secrets written to Vault are encrypted and then written to backend storage. Therefore, the backend storage mechanism never sees the unencrypted value and doesn’t have the means necessary to decrypt it without Vault.

Version 2 provides secret versioning and utilizes a different API compared to version 1. KV v1 is more performant than v2 since there are fewer storage calls due to the lack of additional metadata or history being stored. Migration from v1 to v2 is possible but requires that the secret engine be taken offline during the migration. The process could potentially take a long time depending on the amount of data stored. In general, the additional features provided by KV v2 offset the additional write and storage overhead. Therefore, KV v2 is recommended by default unless you have specific performance requirements.

Note the migration cost: taking the engine offline for a duration proportional to data volume. That makes v1-vs-v2 a decision to get right at enablement, not later.

KV v2 retains a configurable number of secret versions, enabling older versions’ data to be retrievable in case of unwanted deletion or updates. In addition, you can use its Check-and-Set operations to protect the data from being overwritten unintentionally. By default, the kv-v2 secrets engine keeps up to 10 versions.

Delete semantics — three states, not two: the kv v2 plugin uses soft deletes to make data inaccessible while allowing data recovery. When an entry is permanently deleted, Vault purges the underlying version data and marks the key metadata as destroyed.

vault kv delete -mount=secret my-secret                    # soft delete latest
vault kv undelete -mount=secret -versions=2 my-secret      # restore
vault kv destroy -mount=secret -versions=2 my-secret       # permanent
vault kv get -mount=secret -version=1 my-secret            # read a prior version

A customer asked to prove a secret was erased needs destroy, not delete.

Vault static secrets are laid out like a virtual filesystem. The path where you enable the KV secrets engine acts as the root of the file system. Everything after the root is a key-value pair to write to the secrets engine.

The CLI shows this explicitly. Writing to my-secret on a mount named secret reports the Secret Path as secret/data/my-secret, and HVD’s policies are written against that rewritten path:

path "secret/data/billing-service" {
  capabilities = ["create", "read", "update", "delete"]
}

So a policy written against the CLI-visible path (secret/billing-service) grants nothing. This is the most common KV policy mistake in the field.

Before writing any data to the KV secrets engine, it is important to consider how to organize the data under the secrets engine mount. The KV structure is important because it helps to simplify and reduce the number of Vault policies by grouping similar data under the same path.

Antipattern 1: paths based on environments

For example secret/my-app/dev/*, secret/my-app/uat/*, secret/my-app/prod/*. We do not recommend creating paths based on environments. Customers wanting to have Vault in different environments should run a separate Vault instance in each environment.

If this is not feasible due to cost and overhead, you should at a minimum separate prod and non-prod. The benefits: have a process for letting devs add secrets in lower environments, and maintain the same path structure across all environments, which minimizes need for application code changes.

The caveat to raise if they insist on one cluster: consider the increased load on your production cluster from non-prod clients, which might require you to provision hardware with better compute and storage performance.

Antipattern 2: paths based on teams

We also do not recommend creating paths based on teams. This antipattern tightly couples application code to organizational topology, which may prove confusing in the event of team restructuring. We recommend instead to design paths based on the application concern.

Structure your paths as: <app-name>/<service>/<component>/secret

For example, in a webstore app, you might have a billing service and an identity service, and within them you have secrets for specific components:

webapp/billing/checkout/<secret1>
webapp/identity/password-reset/<secret1>

This pairs with the namespace guidance in Unit 4.2 — namespaces align to org structure, paths align to application concerns. Reorganizations then move namespaces, not application code.

The recommendation

Mount a single KV v2 secrets engine with sub-paths per application concern.

Benefits: it reduces the potential of hitting mount table limits, and it reduces operational complexity by centralizing all static secrets under a single mount.

There is a limit to the number of secret engines you can mount. For integrated storage, the maximum number of secret engine mount points is approximately 14000.

Note the contrast with dynamic secrets. Unit 7.5 recommends one secrets engine mount point per database, per application — the opposite shape. The reasoning differs: dynamic mounts each carry a distinct root credential and lease lifecycle worth isolating, whereas KV mounts carry no credentials and multiplying them only consumes mount table entries. Don’t generalize either rule across engine types.

export VAULT_NAMESPACE=<tenant namespace>
vault secrets enable -version=2 -path=secret kv

Or with a description:

vault secrets enable -version=2 -path="kvv2" -description="Demo K/V v2" kv

Vault supports a command line interface, application programming interface, web based user interface, HashiCorp Terraform, and many other configuration management tools such as Ansible.

HVD’s end-to-end example, worth following verbatim because it wires policy to both human and machine auth:

Step 1 — write a secret:

vault kv put -mount=secret billing-service foo=bar

Step 2 — owner policy:

vault policy write billing-service-owner -<<EOF
path "secret/data/billing-service" {
  capabilities = ["create", "read", "update", "delete"]
}
EOF

Step 3 — developer/application policy:

vault policy write billing-service-dev -<<EOF
path "secret/data/billing-service" {
  capabilities = ["read"]
}
EOF

Step 4 — associate with human auth. OIDC uses an external group plus group alias (as in Unit 5.3); LDAP is a direct mapping:

vault write auth/ldap/groups/billing-service-owner policies=billing-service-owner
vault write auth/ldap/groups/billing-service-dev policies=billing-service-dev

Step 5 — associate with machine auth:

vault write auth/aws/role/billing-service-dev auth_type=iam \
  bound_iam_principal_arn=arn:aws:iam::123456789012:role/aws-ec2role-for-vault-authmethod \
  policies=billing-service-dev token_ttl=8 max_ttl=8

vault write auth/approle/role/billing-service-dev \
  secret_id_num_uses=1 \
  secret_id_ttl=15m \
  token_policies="billing-service-dev" \
  token_ttl=1h \
  token_max_ttl=1h

The application is responsible for renewing the Vault token once it authenticates using the Secret ID. Same lifecycle constraint as Unit 6.5.

Step 6 — validate, checking that identity_policies (OIDC, via group) or token_policies (LDAP, direct) carries the expected policy, then confirming the owner can write and the developer can only read.

HVD’s own illustration of the access chain: a secret object is created at secrets/billing-service; a policy named billing-service-read is created where read capabilities are defined against that path; the identity group billing-service-dev is mapped to the policy in the OIDC auth method; as a member of that group, Bob authenticates using OIDC, and the token Bob receives has the policy attached; Bob is allowed to read the secret.

Principle of least privilege. Err on the side of caution if a consumer is uncertain or lacks clarity regarding their access and policy requirements. If a client realizes the need for access to a secret at a later point, it can be identified and addressed accordingly. In contrast, granting access to an unnecessary secret can be challenging to detect and carries potential risks.

That asymmetry — under-granting is self-correcting, over-granting is invisible — is the argument to use when a team pushes for broad access “to avoid tickets.”

Role-based policies. Create several core policies based on functional roles when onboarding users:

• Vault cluster administrator: full access to Vault.
• Vault operator: responsible for general Vault operations such as configuring auth methods, secrets engines, and policies.
• Security team: consume audit logs, review and approve policy changes.
• Developer: read-only access to specific paths required by their applications.
• Application owners: write access to specific paths for their applications.

Beyond these, each application may have distinct requirements that warrants the creation of individual policies. Moreover, multiple policies may be crafted for various stages in the software development lifecycle for each application. Furthermore, automation workflows, such as CI/CD pipelines or other application build tools, may also require specific policies.

Policy templates. Being as restrictive as possible in your policies can lead to a large number of policies to manage. To simplify this process, we recommend the use of ACL Policy Path Templating. Policy templating allows for a much smaller set of policies to be used, but still provide the fine grained access controls needed. (Subject to the 2.0.1 glob restriction — Unit 5.4.)

Version control. Put your policies in a version control system such as GitHub and have a peer-review process for both security and audit purposes. Change management and standardization becomes much easier when policies are centrally managed in a code repository. Vault operators or application teams can submit changes using pull requests and Vault administrators or security teams can approve or deny policy changes. Terraform is the recommended mechanism, consistent with Unit 4.4.

With the initial cluster configuration completed and auth methods enabled, you are now ready to enable your first use case. While Vault has many capabilities around cryptography and secrets management, the initial use-case is often the manual creation of static secrets and secure storage and retrieval.

What is important to consider is the workflow around the lifecycle of the secrets, namely who or what has permission to create, read, update, and delete any particular secret.

HVD assumes at this point you have one human auth method (OIDC or LDAP) and one machine auth method (AWS or AppRole) enabled — exactly the bootstrap sequence in Unit 3.4.

Static secrets are consumed through Vault Agent templates, VSO’s VaultStaticSecret (Unit 9, with type: kv-v1 or kv-v2 and mount specified separately from path), or direct API access.

Two Enterprise capabilities specific to static secrets:

• VSO instant updates — immediate Kubernetes Secret updates via Vault Events, only for VaultStaticSecret and only with Vault Enterprise 1.16.3+. Keep refreshAfter set regardless.
• Secrets Sync — pushes KV secrets to external secret managers. See Unit 6C.

For the consumer-side workflow, see the HVD User Guide page on static secrets.