Unit 7 — Dynamic Secrets

Configure Vault with the appropriate secrets engine and auth mechanisms → request dynamic credentials specifying the desired service and parameters → Vault generates temporary credentials and associates them with a lease determining their lifetime → use credentials while the lease is active → renew or revoke — the application can renew before expiry, or credentials can be revoked manually or automatically at lease expiry.

Plugin ecosystem: cloud credentials (AWS, Azure, GCP); database credentials (Postgres, MySQL, Oracle, MongoDB, RabbitMQ); certificates (PKI); API credentials (Kubernetes, Nomad, Consul, Terraform Cloud); identity management (SSH, LDAP, Kerberos, Okta, GitHub); ecosystem credentials (Artifactory, Venafi, third-party plugins).

Why it matters: since every service is accessing the database with unique credentials, it makes auditing much easier when questionable data access is discovered. You can track it down to the specific instance of a service based on the SQL username. And Vault makes use of its own internal revocation system to ensure that users become invalid within a reasonable time of the lease expiring.

Prioritize dynamic secrets adoption in a phased approach:

1. Cloud credentials in HCP Terraform — leverage Vault’s dynamic provider credentials.
2. Cloud credentials in jobs, CI/CD pipelines — transition embedded credentials to dynamic.
3. DB credentials in jobs, CI/CD pipelines — convert static DB credentials to dynamic.
4. DB credentials in applications — leverage the Vault Agent for an application that is not Vault-aware.
5. PKI — drive certificate signing and renewals.
6. Third-party integrations where needed.

Note this sequences by workload type, starting with the highest-leverage, lowest-friction wins. Pipelines before applications, cloud before database. CI/CD credentials are usually already owned by the platform team, requiring no application-team negotiation.

Dangling leases should be avoided, due to memory consumption involved in lease management and cleanup. Tune your TTLs. Start with a generous TTL value (24-48 hrs) and fine-tune based on the application requirements and lifecycle.

Ensure batch and pipeline jobs’ TTL threshold exceeds the maximum potential of long-running jobs.

Short-lived applications should have short-lived TTLs. Long-lived leases for short-lived applications can have negative memory consequences for Vault.

The enemy is mismatch, not length. A short TTL on a long batch job causes mid-run credential expiry — an outage. A long TTL on ephemeral workloads causes lease accumulation — a memory problem.

(See §13.1 for a tension between this and the monitoring guidance.)

The “root” user is a superuser service account created in an external service exclusively for Vault to manage roles. For AWS, an IAM user and access key; for PostgreSQL, a database superuser and password.

• There must be a 1:1 relationship between “root” users and dynamic secrets configurations in Vault.
• Once added to Vault, this “root” user must not be used or managed outside of Vault.
• It must not be used for more than one dynamic secrets engine within Vault.
• Do not use the same service account for a “root” user in a secrets engine and the “root” user in an auth method.
• Do not use the same “root” credential in more than one location, even within the same Vault cluster.

The “root” user should have only the permissions it needs to create service accounts with their further-reduced scope. These permissions are managed in the external service. Define the appropriate RBAC and account creation statements for your root and dynamic users with the help of your identity and access team and a cloud platform SME or database SME.

rotate-root replaces the manually-entered credential with one that only this Vault configuration knows. After Vault rotates the credential, there is no way to retrieve it. Immediate root credential rotation reduces risk of someone else accessing and using that credential.

vault write -force database-app1/rotate-root/postgresql

vault write database-app1/config/postgresql \
  plugin_name=postgresql-database-plugin \
  allowed_roles=pg-ro,pg-rw,pg-static \
  connection_url="postgresql://{{username}}:{{password}}@$POSTGRES_URL/demoapp?sslmode=disable" \
  username=db_admin_vault \
  password=insecure_password

Parameters are defined by the database plugin being used, so they may differ between databases. The name used here must be re-used by the roles that leverage this connection.

Mount organization. You can configure multiple database plugins per mount point by writing different configs with distinct connection parameters; use allowed_roles to restrict which plugin is used by each role. Consolidating can consolidate configuration of Vault policies for an app that uses multiple database connections. However, providing multiple database connections under one mount point could lead to inadvertent mismanagement, such as revoking credentials across multiple databases accidentally, or requesting the wrong credential altogether. It also could add forensic complexity during an audit.

It is generally a best practice to create one secrets engine mount point per database, per application.

Dynamic roles

Roles define the credential generation. You can have multiple roles per database configuration. Creation statements should be determined in conjunction with your database administration, security, and IAM teams.

tee readonly.sql <<EOF
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
COMMENT on ROLE "{{name}}" IS 'Role managed by Vault';
GRANT CONNECT ON DATABASE demoapp TO "{{name}}";
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";
EOF

vault write database-app1/roles/pg-ro \
  db_name=postgresql \
  creation_statements=@readonly.sql \
  default_ttl=1h max_ttl=24h

db_name must match the name of the database secrets engine object created earlier (not the database hostname).

Username customization improves visibility — for example, you can prepend roles with ‘vault_managed’ to make it more clear to database engineers which roles are managed exclusively via Vault. This can help prevent accidental deletion of Vault’s managed accounts.

Add this to every design. It costs nothing and removes an entire class of incident.

Password policies enable you to meet your organization’s complexity requirements for compliance. Password policies can be defined at the database configuration level, and the role level, enabling the ability to inherit or override the password policy for each database role.

Generation and inspection:

vault read database-app1/creds/pg-ro
vault list sys/leases/lookup/database-app1/creds/pg-ro
vault lease lookup database-app1/creds/pg-ro/<lease_suffix>

Revocation, in increasing blast radius:

vault lease revoke database-app1/creds/pg-ro/<lease_suffix>   # one lease
vault lease revoke -prefix database-app1/creds/pg-ro           # all leases under the role
vault lease revoke -prefix database-app1/creds/                # ALL dynamic roles under the mount

That third command is why consolidated mounts are risky.

Static roles

Static roles are used to rotate the secret password component of a database login credential, while maintaining the same username. Using static roles enables Vault operators to rotate passwords without needing to create and delete users from the database. This can be important for performance at scale, since creating and deleting users can cause table locks impacting database performance. Static roles can be rotated based on a cron-like schedule, and limited to a window of time.

That performance rationale matters more than the compatibility one. A DBA team resisting dynamic secrets on performance grounds is often right, and static roles with scheduled rotation is the answer rather than a compromise.

The static database role must already exist in the remote database. Vault will immediately rotate the password when the role is configured. The Vault “root” user must have RBAC that allows it to ALTER the static role to change the password.

tee rotation.sql <<EOF
ALTER USER "{{name}}" WITH PASSWORD '{{password}}';
EOF

vault write database-app1/static-roles/pg-static \
  db_name=postgresql \
  rotation_statements=@rotation.sql \
  username="vault_static_app1" \
  rotation_period=86400

vault read database-app1/static-creds/pg-static

Role types generally: dynamic roles are used to manage the full lifecycle of credentials in external systems and are created and destroyed by Vault; static roles are used to manage and rotate only the secret portion of a persistent credential, created in the external system first and then added to Vault. All built-in database secrets engine plugins support both dynamic and static roles. LDAP, AWS, and GCP secrets plugins also provide static roles. The Azure secrets engine has the option to use existing service principals, although that plugin calls them application_object_id.

Onboarding friction: when a static database user is onboarded, by default Vault immediately and automatically rotates the database user’s password. This immediate rotation can add additional operational overhead to the onboarding process and has proven challenging for some organizations. Configuration options exist, including disabling the automatic rotation of static role passwords during Vault onboarding. Confirm the parameter name and per-plugin support against the database secrets engine documentation before writing a customer runbook — for a large legacy estate, uncontrolled immediate rotation at onboarding is an outage.

Rootless connections

PostgreSQL supports using static roles and password rotation with a rootless DB connection configuration. In this workflow, a static DB user can be onboarded onto Vault’s static role rotation mechanism without the need of privileged root accounts to configure the connection. Instead of using a single root connection, multiple dedicated connections to the DB are made for each static role. This workflow does not support dynamic roles/credentials.

It is highly recommended that the DB users being onboarded as static roles have the minimum set of privileges. Each static role will open a new connection into the DB. Granting minimum privileges ensures that multiple highly-privileged connections to an external system are not being made.

Out-of-band password rotations will cause Vault to be out of sync with the state of the DB user, and will require manually updating the user’s password.

Oracle likewise does not support dynamic roles/credentials with rootless DB connections, configured with self_managed=true.

This is the answer when a customer’s DBA team refuses to give Vault a superuser account.

Network topology

For dynamic secrets to be implemented, Vault needs a network path to the resource being managed. Where the cluster is in a shared platform-team account and business units run databases in isolated subnets, Vault needs access to each of the db subnets.

When a Vault DR/PR cluster is available, ensure that the secondary clusters have access to the tenant databases as well. This will ensure that in case of a failover, the DR/PR cluster is able to manage dynamic secrets database accounts.

The trap: in case of region failure and the DR Vault cluster and the DR DB is promoted, due to the way the DB mount is configured, the Vault DR cluster may not have a way to access the DR DB. We recommend that a separate DB mount is created for DR purposes. This assumes the application is configured to access the DR DB mount and can request new leases.

The cleaner alternative: the connection parameter in the database secrets engine config is the key to reaching the database from Vault. By using the same client-facing FQDN or IP address for all failover replicas of your database, you can avoid configuring multiple mounts for database failover. Configure your database replication with Virtual IP addresses, global load balancer, or DNS with failover.

Discovery question: “when your database fails over, does its connection string change?” Most customers haven’t considered it.

Other database types

IBM Db2 — access to Db2 is managed by facilities that reside outside the Db2 database system. By default, user authentication is completed by a security facility that relies on operating system based authentication. This means that the lifecycle of user identities in Db2 aren’t capable of being managed using SQL statements, which creates an operational challenge for credential lifecycle management. To provide flexibility, Db2 ships with authentication plugin modules for LDAP.

AWS DynamoDB — not supported by the database secrets engine; use the AWS secrets engine to provision dynamic credentials capable of accessing DynamoDB.

There is a full validated pattern: Integrate Vault with z/OS RACF for secrets management (authors: Carl Hovi, Andy Baran, Justin Jarae).

What it does: configure Vault to manage secrets used to access resources on IBM z/OS mainframe systems secured by RACF. Organizations running mainframe workloads can use Vault to make RACF passphrases available securely to applications running anywhere, including on distributed and cloud platforms, and to automatically rotate RACF passphrases and passwords.

Mechanism: Vault’s LDAP secrets engine to integrate with RACF through the IBM LDAP SDBM interface provided by z/OS. The LDAP interface provides the integration point between Vault and RACF without requiring modifications to z/OS systems.

Workflow: Vault connects to z/OS through LDAP using a privileged RACF administrative account; Vault stores and rotates the administrative account’s passphrase; applications authenticate to Vault; Vault uses the administrative account to rotate RACF credentials on a defined schedule; applications retrieve current credentials.

Prerequisites: Vault Enterprise 1.20.4 or later with namespace support or HCP Vault Dedicated; network connectivity from Vault to the z/OS LPAR; a z/OS LPAR running RACF with RACF passphrase support enabled; the IBM z/OS LDAP server SDBM LDAP interface enabled; a RACF administrative account with permission to change a passphrase of other RACF accounts with the NOEXPIRED option; access to a 3270 emulator.

The NOEXPIRED gate

When administrators manually update RACF passphrases, they typically set them as EXPIRED, requiring users to change them at next login. However, Vault needs to set application credentials with the NOEXPIRED option since the applications which use these credentials cannot interactively change these passphrases. This is why the RACF account that Vault uses needs to be able to update the passphrases of other RACF accounts with the NOEXPIRED option.

The failure signature:

LDAP Result Code 80 "Other": ICH21005I NOT AUTHORIZED TO SPECIFY
PASSWORD/NOPASSWORD, OPERAND IGNORED.
ICH21005I NOT AUTHORIZED TO SPECIFY NOEXPIRED, OPERAND IGNORED.

Messages starting with ICH are from RACF (not from Vault).

The passphrase update performed through ALTUSER is the same action Vault performs to other RACF accounts through the z/OS LDAP interface. Use that framing with RACF admins — you’re automating ALTUSER, not introducing a new mechanism.

Credential types

RACF supports traditional passwords limited to 8 characters or less with restricted character sets, and passphrases supporting 9-100 characters with enhanced complexity requirements. For enhanced security, use RACF passphrases. The RACF administrative account that Vault uses must only have a passphrase assigned (not a traditional password) because of its elevated permissions.

Traditional RACF passwords can be up to 8 characters, using only upper case A-Z, 0-9 and @ $ #. RACF passphrases can be up to 100 characters and add lower case and many special characters.

Passphrase policy — and the field-hardened character exclusions

length=14
rule "charset" {
  charset = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
  min-chars = 1
}
rule "charset" {
  charset = "abcdefghijklmnopqrstuvwxyz"
  min-chars = 1
}
rule "charset" {
  charset = "0123456789"
  min-chars = 1
}
rule "charset" {
  charset = "@$#!?.;&<>()[]{}~*_|="
  min-chars = 1
}

vault write -namespace=admin/z sys/policies/password/racf_passphrase_policy \
  policy=@/path/racf-passphrase-policy.hcl

A 53-character variant exists for customized z/OS rules.

Why +, -, ^, %, ¢ are omitted: under some 3270 emulators, TSO logins fail if the passphrase includes the character ^, but that passphrase works in an ldapsearch end user bind. Some have seen login attempts fail if a RACF passphrase ends with the - or + character. The % character has special meaning when z/OS MFA appears. ¢ because most keyboards lack the key.

The operational warning: when an engineer first tests Vault rotating RACF passphrases, they might use a 3270 terminal emulator that has a problem with certain special characters. If a RACF passphrase does not work in their 3270 emulator, they might think Vault is not working properly. If a tester is not well connected to their company’s z/OS mainframe team, or if they are a novice about z/OS and RACF, this can compound the confusion.

If including a specific character in RACF passphrases can cause a problem in non-production tests then avoid that character, even if it works in the planned production usage.

The default RACF passphrase policy requires at least 14 characters, at least 2 alpha characters, at least two numeric/special characters, and does not allow 3 consecutive identical characters. If your z/OS system enables the default IBM rule, it rejects a new passphrase value that includes three consecutive identical characters, and Vault retries with a newly generated passphrase. IBM has released an option for z/OS 3.1 and 3.2 (APAR OA67750) which allows disabling the IBM default RACF passphrase syntax rules.

You can change the RACF passphrase syntax rule used by Vault at any time without needing to rebuild the secrets engine definition. Note that if you do this there is no immediate check performed by Vault to verify that the new rule causes Vault to create valid RACF passphrases.

Configuration

vault namespace create admin
vault namespace create -namespace=admin z

vault secrets enable -namespace=admin/z -path=admin/z/ldap-zracf-passphrase ldap

vault write -namespace=admin/z admin/z/ldap-zracf-passphrase/config \
  binddn='racfid=HVLTADM,profiletype=user,cn=tec2racf,o=ibm,c=us' \
  bindpass=xXxXxXxXxXxXxX \
  url=ldap://tecz22.example.com:389 \
  schema=racf \
  credential_type="phrase" \
  password_policy=racf_passphrase_policy

credential_type specifies phrase, which instructs Vault to update RACF account passphrases instead of updating RACF account passwords.

Your mainframe team needs to tell you the full RACF account DN. All RACF accounts on a single z/OS mainframe share the same DN syntax after the account name. Ask for that string in discovery — one line that unblocks configuration.

vault write -f -namespace=admin/z admin/z/ldap-zracf-passphrase/rotate-root

For security reasons it is important to have Vault perform an initial rotation of the RACF administrator passphrase as soon as possible. For anything other than an initial sandbox test, do this first. The initial rotation is a one-time step that cannot be undone. After you perform this step, only Vault knows the RACF passphrase for this administrative account. Your z/OS security administrators do not have access to it.

vault write -namespace=admin/z admin/z/ldap-zracf-passphrase/static-role/javaappserv5 \
  dn='racfid=HVLTDB5,profiletype=user,cn=tec2racf,o=ibm,c=us' \
  username='javaappserv5' \
  rotation_period="24h"

vault read -namespace=admin/z admin/z/ldap-zracf-passphrase/static-cred/javaappserv5

If the command succeeds without error, Vault has demonstrated it has the RACF authority to rotate this secret.

The vault read command output uses the label “password” whether Vault is rotating longer RACF passphrases or traditional RACF passwords.

The cascading-expiry risk

If for any reason something goes wrong with Vault’s ability to rotate this RACF account’s passphrase, you want the opportunity to discover this failure through log messages well before this RACF account’s passphrase expires. If this RACF account’s passphrase expires, it cannot rotate the passphrases of other RACF accounts, and then their passphrases also expire, which could cause widespread operational issues. You must put monitoring in place to detect if there is ever a problem in rotating Vault’s root credential for RACF.

If a particular z/OS mainframe requires updating passphrases every 90 days, configuring Vault to rotate its root credential every 20 days is a reasonable choice. LISTUSER output includes PASS-INTERVAL (required change frequency) and PHRASEDATE (last change date).

Application policy — the correct scoping model

path "admin/z/ldap-zracf-passphrase/static-cred/javaappserv5" {
  capabilities = ["read"]
}

Caveats in the pattern’s own examples: it uses userpass for instructional purposes and recommends a more secure method that leverages the organization’s preferred authentication tool or directory, naming AppRole or Kubernetes authentication. It also uses a broad namespace admin policy and notes the elevated permissions granted include the ability to enable any authentication method into this namespace. Your organization might not want to grant this. Don’t copy either into a customer environment.

Design note: the LDAP auth method and the LDAP secrets engine both need a bind account. Per §7.4, do not share a service account between them — the secrets engine rotates its bind credential to an unknowable value, and auth breaks at first rotation.