Unit 3 — Bootstrapping a Cluster

When a Vault server is first initialized, no auditing is enabled. Audit devices must be enabled by a root user using the CLI, API, or Terraform.

Why first: an otherwise-successful request will fail if it cannot be logged to at least one configured audit device. Failure to log to at least one audit device will prevent Vault from servicing requests. This is by design to ensure all requests and responses are captured correctly.

The three device types

File — writes logs to a file, appending. Does not support log rotation; the operator must use third-party tools such as logrotate. Sending SIGHUP to the Vault process causes file audit devices to close and re-open their underlying file. It is important to rotate and archive audit log files to prevent growth that consumes the entire disk. Vault will not respond to any API requests if there is a blocked file audit device.

Syslog — does not support remote syslog destinations and always sends to a local syslog agent.

Socket — writes to TCP, UDP, or UNIX socket. Due to the unreliable nature of the underlying protocol, do not enable the socket audit device unless absolutely necessary. If you do, always enable a secondary non-socket audit device.

Why two devices, and the two failure modes

Enable at least two audit devices of different types.

Improved availability:

• A blocking failure is one where an attempt to write stalls without returning an error. This is unlikely with a local disk device, but could occur with a network-based audit device.
• A non-blocking failure is one where an attempt to write returns an error and no audit log is written.

With multiple devices, if any fail in a non-blocking fashion, Vault requests can still complete successfully provided at least one device successfully writes the record. If any device fails in a blocking fashion, Vault requests will hang until the blocking is resolved.

Checking and verification: set up one audit log for analysis, and another for secure storage and archiving. The archival device should write to a filesystem or syslog destination configured with strict access control permissions. Read-only access can be granted where reconciliation is needed, but otherwise these logs remain untouched, ensuring an unaltered version exists for security review.

Configuration

export VAULT_ADDR=https://<vault FQDN>:8200
export VAULT_TOKEN=<your root token>

vault audit enable file file_path=/vault/vault-audit.log
vault audit enable syslog tag="vault" facility="AUTH"

Recommendation: enable a file audit device as well as a syslog audit device.

Permissions: the Vault process user needs write access for file, and capabilities such as CAP_SYSLOG for syslog.

Replication behavior: audit device configuration is replicated to all nodes within a cluster by default, and to performance/DR secondaries. Each node writes to its own audit log, in the same locations as the active node. Before enabling a device, ensure all nodes within the cluster(s) — including DR and performance secondary clusters — will be able to successfully log to it, to avoid Vault being blocked from serving requests. A device can be limited to only the nodes within the cluster using the local parameter.

Audit logs from all nodes need to be analyzed to audit any event, so it is best practice to use a centralized logging solution.

Sizing and the UDP trap

A typical audit log entry can be 1kb–3kb, meaning a node servicing 10,000 requests an hour can write 10–30mb of data. Use log rotation, transfer to external storage, and configure the audit logs to write to a separate logical volume to avoid disk IO contention with Vault’s internal storage when using integrated storage.

Audit messages for some operations can be larger than a maximum-size single UDP packet. Because UDP is connectionless, if a message is larger, that audit log message will fail silently — Vault will have no knowledge that the message was too large. If possible, configure a TCP listener; because TCP is connection-oriented, Vault will have awareness whether syslog messages were successfully received. This can result in a blocked audit device if TCP connections are unsuccessful. To avoid this possibility, consider using a file backend and having syslog configured to read entries from the file.

Error signatures

* sanity check failed; unable to open "/vault/vault-audit.log" for writing: open /vault/vault-audit.log: permission denied

[ERROR] enable audit mount failed: path=syslog/ error="Unix syslog delivery error"

Unix syslog delivery error can mean the syslog service is not enabled on the host or that Vault cannot access it — often due to SELinux restrictions. Temporarily set permissive mode with setenforce to test.

Hashing

Each line in the audit log is a JSON object containing all the information for a given request and corresponding response. By default, sensitive information is hashed before it is logged. Most strings are hashed with a salt using HMAC-SHA256, so secrets are not in plaintext. You can still check the value of secrets by generating HMACs yourself using the /sys/audit-hash API endpoint.

Audit device logs are separate and unrelated to Vault operational logs.

By default Vault includes no administrative policies. A new installation has only default, which grants no access to functional resources.

Operator policy — root namespace. For operators with full access, responsible for cluster level configurations and any requests to the restricted API that requires the root namespace:

# * (glob) character matches any prefixes in the path and can only be used at the end
path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

export VAULT_TOKEN=<your root token>
vault policy write operator operator-policy.hcl

Not needed for HCP Vault, where the root namespace is inaccessible.

Tenant admin policy — org/tenant namespace. Tenant administrators are responsible for day to day Vault operations for their Vault tenant (a parent namespace and a number of subordinate application/environment namespaces). They only have access to their tenant namespace and below and do not have access to the root namespace.

# List existing policies
path "sys/policies/acl" { capabilities = ["list"] }

# Create and manage ACL policies
path "sys/policies/acl/*" { capabilities = ["create", "read", "update", "delete", "list"] }

# Manage auth methods broadly across Vault
path "auth/*" { capabilities = ["create", "read", "update", "delete", "list", "sudo"] }

# Create, update, and delete auth methods
path "sys/auth/*" { capabilities = ["create", "update", "delete", "sudo"] }

# List auth methods
path "sys/auth" { capabilities = ["read"] }

# Managing identity
path "identity/*" { capabilities = ["create", "read", "update", "delete", "list"] }

# Enable and manage the key/value secrets engine at `secret/` path
path "secret/*" { capabilities = ["create", "read", "update", "delete", "list"] }

# Allow managing leases
path "sys/leases/*" { capabilities = ["read", "update", "list"] }

# Manage namespaces
path "sys/namespaces/*" { capabilities = ["create", "read", "update", "delete", "list"] }

# Manage secrets engines
path "sys/mounts/*" { capabilities = ["create", "read", "update", "delete", "list"] }

# List existing secrets engines.
path "sys/mounts" { capabilities = ["read"] }

# Configure License
path "sys/license" { capabilities = ["create", "read", "update", "delete", "list"] }

# Configure Vault UI
path "sys/config/ui" { capabilities = ["read", "update", "delete", "list"] }

This example policy is only a subset of Vault’s API. If additional administrative capabilities are necessary, edit the admin policy when the need arises.

Note that this grants sys/namespaces/* and auth/* with sudo — deliberate, for tenant self-service. It’s why the Sentinel governance layer in Unit 5 exists.

It is considered a best practice not to persist root tokens. Use the root token only for just enough initial setup. Once you enable an auth method with appropriate policies allowing Vault admins to log in and perform operational tasks, the admins should use the auth method to authenticate instead of using the root token.

Then, explicitly: after you have validated that you can login using your first auth method, revoke the root token to eliminate the risk of exposure.

vault login <your root token>
vault token revoke <your root token>

Root token properties before revocation: token_duration ∞, token_renewable false, policies ["root"].

In case of an emergency when a root token is absolutely necessary — for example, loss of auth method preventing admin access requiring break glass access — generate a new root token using the operator generate-root command.

“Do you still have your initial root token?” should be answered no. That’s an assessment question worth asking every customer.

1. Initialize and unseal (with PGP-encrypted key shares and root token).
2. Enable two audit devices of different types.
3. Create the operator policy in the root namespace.
4. Create the admin namespace and the tenant admin policy.
5. Enable the first human auth method in the root namespace for operators.
6. Enable a human auth method in the org namespace for tenant admins.
7. Validate login via the auth method.
8. Revoke the root token.