Unit 10 — Day-2 Operations

Vault provides metrics, operational logs, audit logs, and a health API. Combined with host monitoring, use all of these elements to gather an accurate picture of the security, compliance, and performance of your Vault environment.

To get a full picture and observe any anomalies it is important to profile operational monitoring of Vault over time. Monitor all metrics continuously and implement tested alerting thresholds. Establish baselines and trends, with any deviation from them an indication of a potential problem.

HashiCorp cannot provide definitive metric values in every case, so detecting deviations from a moving average is the most reliable indicator. In order to define a mean value, isolate the initial sample value for each metric and then monitor this over a day and a normal business week to establish a reasonable starting point. Those mean values can change over time as your usage changes.

Vault does not offer a solution for aggregating, visualizing, or alerting on telemetry or audit data. Recommended: Prometheus and Grafana, or Splunk.

The full tables are on the HVD Monitoring and Observability page. These are the ones with hard thresholds or high diagnostic value.

Availability:

Host:

Runtime:

Interpretation notes worth memorizing:

• Memory usage is constant when your system is in a steady state, as there is a balance of lease issue and revocation. If you see memory usage starting to grow, then begin investigating. Memory usage does not spike — it grows in line with an increase in the workload. If it grows more quickly, then this is an indication that something is wrong.
• Encryption can place a heavy demand on the CPU. Particularly important if the Transit or the Transform secrets engines are in use.
• A consistently low vault.core.leadership_lost value is indicative of high leadership turnover and overall cluster instability. Spikes in vault.core.leadership_setup_failed indicate that your standby servers are failing to take over as the leader.
• If you observe a spike in vault.core.handle_login_request but little to no increase in the number of tokens issued (vault.token.creation), examine the cause immediately. If clients cannot authenticate to Vault quickly, they may be slow in responding to the requests they receive, increasing the latency in your application.
• A large and unexpected delta in vault.expire.num_leases can indicate a bulk operation, load testing, or runaway client application generating excessive leases and requires immediate investigation.
• As the number of entities rises the more taxing additions and updates to entities and groups on the overall system become. To remediate long times required for those updates, you can consider using smaller internal groups or using external groups.
• The higher the number of policies, the higher the time it takes to assemble the access control list for a token (vault.core.fetch_acl_and_token).
• vault.wal.write_controller.reject_fraction is connected to Vault’s write overload protection. Consistently high fractions could indicate that there is a hardware failure or that the initial sizing has been outgrown by the actual workloads.
• Increased time spent in Merkle diff operations can be indicative of a larger problem in determining differences between data in the two clusters.

The general purpose of the audit log is for troubleshooting events that have adversely affected operations, rather than as an ongoing monitoring tool.

Configure an alert to notify if continuous addition to the audit log endpoints stops for any reason, especially during normal business hours and maintenance windows.

Use audit logs to define baseline usage, alert on high deviations, and therefore be able to detect potential misconfigurations in applications consuming Vault. For example, if on a typical day there are 500 authentications with a token and 100 reads of a KVv2 secrets engine within a namespace, and suddenly there are 100,000 reads, there is probably an application misconfigured somewhere or a new usage pattern.

Observable usage patterns to build detections for: use of the root token; creation of a new root token; increased attempts to access varying invalid paths with the same credentials; increased attempts to access varying paths with insufficient permissions; special tokens with extended permissions used from an unexpected IP address; permission denied (403) responses; invalid username or password; Vault policy modifications; enabling a new authentication method; modification or creation of an authentication method role; use of Vault by people-related accounts outside regular business hours; requests originating from unrecognized subnets; authentication attempts with varying credentials from the same subnets; many authentication attempts in a short time; sealing of Vault; changes to the audit devices; K/V secret create/update/delete; PKI certificate generation and revocation; changes to the Transit Minimum Decryption Version Config; Transit key deletion.

Privileged endpoints — configure alerts for any access:

/sys/policy          /sys/policies        /sys/rekey
/sys/rekey-recovery-keys                  /sys/replication
/sys/audit           /sys/audit-hash      /sys/generate-root
/sys/rotate

Use a SIEM log aggregation service to detect endpoint requests and raise alerts to operational and security staff.

Comparing sensitive values: sensitive information written as JSON strings is HMAC’d by default. This includes error messages. When querying the audit logs for a specific error message you can use the audit-hash API to find the corresponding hash.

vault write /sys/audit-hash/file input="invalid username or password"

Monitor sys/health.

• Configure an alert when a standby node cannot connect to the active node for a prolonged time (status code 474) as this means the node is not part of the cluster.
• Configure an alert if Vault is sealed (status code 503) as it cannot serve any requests in the sealed state.

Transactional monitoring is a way to mimic and test user behavior. While monitoring all other aspects of the Vault service are useful in understanding health and performance, measuring an actual user transaction is particularly of use observability-wise.

#!/bin/bash

# Variables
CONTROL_VALUE="<value you expect the secret to have>"
USERNAME="<username>"
PASSWORD="<password>"
SECRET_KEY="<key>"
SECRET_MOUNT="kv"
SECRET_PATH="test-secret"

# Authenticate to Vault
vault login -method=ldap username="$USERNAME" password="$PASSWORD"

# Write secret to Vault
vault kv put -mount="$SECRET_MOUNT" "$SECRET_PATH" "$SECRET_KEY"="$CONTROL_VALUE"

# Retrieve secret and test
TEST_VALUE=$(vault kv get -format=json -mount="$SECRET_MOUNT" "$SECRET_PATH" | jq -r ".data.data.$SECRET_KEY")

# Compare retrieved value with expected value
if [[ "$TEST_VALUE" == "$CONTROL_VALUE" ]]; then
    echo "SUCCESS: Retrieved value matches the expected secret value"
    exit 0
else
    echo "FAILURE: Retrieved value does not match the expected secret value"
    echo "Expected: $CONTROL_VALUE"
    echo "Got: $TEST_VALUE"
    exit 1
fi

At each step the success or failure can be measured, and the amount of time it took to execute each step can be recorded. This allows the operator to not only understand whether the service is functioning, but also how it is performing, especially when associated with CPU, RAM, and disk utilization statistics. Run at a defined interval via cron. As the script matures, it can be expanded to monitor aspects such as performance replication or dynamic credentials.

Operational logs are configured at trace, debug, info, warn, or error. Configure your monitoring solution to alert on error log events. HashiCorp Technical Support Engineers will often ask for Vault operational logs as a troubleshooting step.

1. Revoke the compromised token:
   bashCopy

   vault token revoke <token_id>
   vault token revoke -accessor <token-accessor>
   

2. Determine the time frame when the unauthorized activity occurred.
3. Search by Token ID — generate the hash and search the audit logs:
   bashCopy

   vault write /sys/audit-hash/file input="$VAULT_TOKEN"
   

4. Review the audit log entries — look for commands or actions like read, write, or delete that involve secret paths or keys.
5. Rotate affected secrets:
   bashCopy

   vault lease revoke database/creds/readonly/<lease-id>
   vault lease revoke -prefix database/creds
   vault read database/creds/role-name
   

Restricting access — two methods:

Seal Vault. Sealing wipes the root key from memory, effectively shutting down all operations in Vault. In multi-tenant environments this is a highly disruptive operation.

vault operator seal

Lock the API of a specific namespace. This stops all operations on a specific namespace and any child namespaces.

vault namespace lock hvd
# note the returned unlock_key
vault namespace unlock -unlock-key <unlock_key> hvd

Namespace locking is the surgical option. Most operators reach for seal because they don’t know this exists.

Snapshots protect your Vault cluster against data corruption or sabotage of which disaster recovery replication might not be able to protect against.

Restated: while typically you would rely on your DR secondary cluster to protect against data loss, there are cases that disaster recovery replication cannot safeguard against, such as data corruption or malicious data manipulation. In these cases, snapshots are the only source of data you will be able to trust when attempting to restore service.

That’s the answer to “we have DR replication, why do we need backups?” — replication faithfully replicates corruption.

Prerequisites: Vault is initialized, keyholders are available with access to the unseal keys for each, you have access to tokens with sufficient privileges for both clusters, and encrypted data is stored in the storage backend.

Automated snapshots on the primary:

vault write sys/storage/raft/snapshot-auto/config/[:name] [options]

Restore procedure (integrated storage only):

1. Prepare and initialize the new restored Vault cluster. Note that these keys and tokens are only temporary; the original unseal keys will be needed following restore.
2. Copy the snapshot file onto the new cluster:
   bashCopy

   vault operator raft snapshot restore -force <backup.snap>
   

   Because the snapshot you are restoring is from a different cluster, the unseal keys of the new cluster will not be consistent with the snapshot data. In cases like this, you must specify the -force flag for the snapshot restore to succeed.
3. Unseal:
   bashCopy

   vault operator unseal <unseal_key>
   

⚠ The -force caveat. You can safely use -force when restoring a snapshot from a system using Shamir seal. After the restore, the original storage entry for the root key will be available, allowing the original cluster’s unseal keys to decrypt it. However, with autounseal, it is crucial to use the same seal configuration for the new cluster as the one used for the cluster being restored. If not, data decryption will not be possible after the restore, and using -force will disable a valuable safeguard.

Read alongside §2.5: auto-unseal creates a hard dependency on the seal mechanism, and that dependency extends to restore. Confirm the customer’s backup runbook states this.

DR replication — full synchronization of configurations, policies, secrets engines, auth methods, audit devices, KV secrets, encryption keys, tokens, and leases. Even local mounts replicate, creating a complete duplicate.

Performance replication — selective shared state, including configurations, policies, secrets engines, auth methods, KV secrets, and encryption keys. Tokens and leases are local to each cluster and do not replicate.

DR implications: tokens and leases replicate fully, so workloads face no disruption during a failover. If a primary fails after generating a credential lease, the promoted secondary has knowledge of that lease and can enforce its TTL, renewal, or revocation. This prevents orphaned credentials. Without DR, a primary failure orphans active leases, leaving certain dynamic credentials exposed indefinitely.

Performance implications: authentication and lease management happen per cluster, so a workload ties to one Vault instance for its session. If that cluster fails, the workload must re-authenticate to another cluster, as its token is not shared. The originating cluster is the only one that can handle renewal or revocation.

The orphaned-lease risk: performance replication alone does not prevent orphaned leases. If a PR cluster fails, its local leases disappear unless the cluster recovers. While some dynamic secrets have target-system enforced TTL values (like AWS STS tokens), others (like database credentials) require Vault to revoke them. Without the failed cluster, these credentials remain active, creating a security risk.

Recommended topology: pair each performance secondary with its own dedicated DR secondary. Treat each performance secondary as a local primary for its workloads. If it fails, you promote its DR counterpart, which inherits all local tokens and leases, ensuring you can manage and revoke credentials.

The budget justification: identity brokering makes Vault a critical runtime dependency for applications and workloads; implementing both forms of replication allows you to achieve the required levels of operational robustness.

Monitoring performance replication should include ensuring that (1) both the primary and secondary clusters are online and in a healthy state, (2) network communication between the primary and secondary clusters has not become disrupted, and (3) the replication subsystem is operating in a healthy state on both clusters and data sync is keeping pace.

For (2): a synthetic check can be run periodically against the sys/replication/performance/status API endpoint on both clusters to check the value of the “state” field. If the value of the state field is “idle” then this indicates that communication may have become disrupted and replication is no longer taking place.

The progress and health of synchronization can be checked by comparing the values of last_performance_wal on the primary and last_remote_wal on the secondary. After writing a piece of replicated data, the last_remote_wal on the secondary should match the last_performance_wal on the primary for a short period of time.

When it comes to upgrading HashiCorp Vault, it is essential to follow well-defined operating procedures to ensure a smooth transition and minimal disruption. Vault upgrades are crucial for security and performance enhancements, but they can also be complex.

Ensure the run book is understood by multiple staff in the Platform Team and tested by someone who is not the author to ensure it is fully understandable.

That test — written by one person, executed by another — is the single best quality check on any runbook, and it applies well beyond upgrades.

On HCP Vault, upgrades are managed by the HCP control plane.

Autopilot

HashiCorp highly recommends that you utilize the autopilot upgrade feature, as it greatly streamlines and simplifies the upgrade process. Autopilot is purpose-built to minimize administrative involvement and reduce downtime. It synchronizes all data to new nodes, and it accomplishes this through snapshots instead of the replication subsystem. This approach is often faster and more comprehensive. Raft seamlessly manages this process.

Another benefit of utilizing autopilot is that there is less of a need to expend resources closely monitoring traffic during upgrades, as this feature comes with many safety checks.

Autopilot ensures that Vault executes parts of the upgrade process in the correct order, resulting in minimal downtime. You can leverage Terraform for setting up and deploying configurations and Packer for creating new nodes from templates.

Two considerations:

Autopilot simplifies the safe transition of leadership to the new nodes, while demoting the old ones so as not to impact quorum. However, this ease of provisioning should be monitored to ensure it aligns with your infrastructure requirements.

Autopilot does not automatically manage the removal of old nodes. You should always manually decommission outdated nodes once an upgrade is successfully completed to prevent resource inefficiencies.

On tuning: the default autopilot configurations are suitable for most scenarios. Unless you have a specific reason for doing so, it is advised against tuning the autopilot config. Picking inappropriate values can lead to difficult-to-debug problems. These default configurations are best suited for relatively static clusters, such as those deployed on manually provisioned VMs. However, if you are operating in a more dynamic environment, like Kubernetes or behind an auto scaling group or equivalent, tuning may be beneficial.

Automated upgrade in an AWS Auto Scaling group

Vault autopilot enables you to automatically upgrade a cluster of Vault nodes to a new version as updated nodes join the cluster.

Prerequisites: basic knowledge of AWS Auto Scaling including scale-in protection; a general understanding of how Vault upgrades work; an ASG containing a Vault cluster running Vault Enterprise 1.11.0 or later; upgrade migration must not be disabled on the cluster. Vault Enterprise enables automated upgrade migrations by default.

vault operator raft autopilot get-config | grep 'Upgrade Migration'

Expected output:

Disable Upgrade Migration             false

Documentation inconsistency to be aware of. HVD gives the re-enable command as vault operator raft autopilot set-config -disable-upgrade-migration=true. Read literally, setting disable-upgrade-migration to true would disable migration, not re-enable it. Verify the intended value against the autopilot API reference before running it in a customer environment.

Procedure:

1. Create a launch template that can create new instances with the desired new version of Vault. This can be accomplished by using an AMI with Vault already installed, or via a userdata script that installs Vault at boot time.

2. Configure the ASG for your Vault cluster to use the new launch template. Once the new launch template is in place, future instances created by the ASG will be created using the new version of Vault. Existing instances will not be affected.

3. Update the ASG by changing the desired capacity count to at least double the current number of nodes. For a 3-node cluster, set desired capacity to 6.

   The number of new nodes running the target version of Vault must be equal to or greater than the number of existing nodes before the autopilot process will begin promoting new nodes to voters.

4. Monitor the upgrade migration:

watch -n 0.5 'curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    $VAULT_ADDR/v1/sys/storage/raft/autopilot/state \
    | jq -r ".data.upgrade_info"'

Continue to monitor the output until the status field reads await-server-removal:

{
    "other_version_non_voters": [
        "vault_2",
        "vault_3",
        "vault_4"
    ],
    "status": "await-server-removal",
    "target_version": "1.12.0.1",
    "target_version_voters": [
        "vault_5",
        "vault_6",
        "vault_7"
    ]
}

other_version_non_voters contains the list of nodes running the old version of Vault, which have been demoted to non-voting status. target_version_voters contains the list of nodes running the new version, which have been promoted to voting status.

5. Remove the nodes running the old version from the cluster:

vault operator raft remove-peer <node id>

6. Temporarily enable scale-in protection for each instance running the new version. This will prevent the ASG from destroying the new nodes during scale-in events.
7. Update the ASG again, reducing the desired capacity count to its original value. Instances with scale-in protection enabled will be protected from termination, while instances without protection will be terminated.
8. Verify with vault status — verify that the version number matches the target version of Vault, and that Sealed is false.
9. Remove scale-in protection from the remaining instances.

Note the shape this shares with the Kubernetes procedure in Lab D: new nodes join first, old nodes are demoted and explicitly removed with remove-peer, and the active node goes last. The mechanism differs; the ordering principle does not.

Vault Enterprise deliberately does not support an automatic failover/promotion of a DR secondary cluster. It is possible to automate accurate detection with your own development/tooling, and triggering/orchestrating failover by leveraging Vault’s Replication and DR APIs. But having an automated means to failover may reduce outage time, but we do not recommend fully automated failover.

Reactive awareness sources: application alerting, container/server logs, APM/tracing tools, Vault logs, Vault telemetry, end-user complaints.

Proactive action: there are situations that your business continuity plan might prescribe a move to a DR environment ahead of the actual event. The example given: many financial organizations failing over to non-coastal datacenters when hurricane Sandy approached the New York area in 2012.

The triage runbook — a pre-authored and pre-tested set of verifications:

• Are all services that use Vault experiencing problems, or one/subset?
• Is Vault itself (logs, telemetry) indicating a problem?
• Is there current maintenance that is possibly causing false alarms?
• Is Vault the actual issue, or could it be downstream (ie, the auth Vault uses to a cloud provider has been revoked)?
• Is the Vault cluster and each node responding if accessed directly vs through the load balancer?
• Is the issue something that would have been replicated to the DR cluster already (ie, accidental mass-revocation of tokens, or configuration change)?
• Will DR resolve this issue? Or will we need to restore from backup?

Promotion. Prerequisite: a Batch DR operation token.

1. Disable the network load balancer to the primary region’s Vault nodes, and disable the global load balancer.
2. Promote the DR secondary cluster to become the new primary. Set VAULT_ADDR to the secondary’s load balancer; verify with vault status; promote.
3. Verify the cluster is acting as a primary, that it has no secondaries, mode is primary, and its state is running:
   bashCopy

   vault read sys/replication/dr/status --format=json
   

   Expect known_secondaries: [], mode: "primary", secondaries: [], state: "running".
4. Configure the global load balancer to target the new cluster. Verify connectivity via browser.
5. Notify Vault consumers and stakeholders Vault is back online.
6. As soon as the old primary is accessible, demote it to prevent split brain scenario.

Failback. Prerequisite: a Batch DR operation token.

1. If you have not yet demoted the old primary, you must do that before bringing it online and resuming replication.
2. Monitor Vault replication to ensure that the WALs are in sync: the state of the secondary is stream-wals; merkle_root on the primary and secondary should match; the last_dr_wal on the primary should match the last_remote_wal on the secondary.
3. Notify Vault consumers and stakeholders of planned outage.
4. Disable load balancer to the current primary, and the global load balancer.
5. Demote the current primary to a secondary.
6. Promote the original region’s DR secondary as the new primary and verify.
7. Establish replication: generate a secondary activation token from the new primary, copy the wrapping_token, and update the primary of the other cluster using the batch DR operation token and the activation token.
8. Enable network and global load balancers.
9. Notify Vault consumers and stakeholders.