Unit 2 — The Reference Architecture

A single Vault cluster comprises: a cluster of Vault nodes, a load balancer, snapshot storage, a seal mechanism, and audit device(s).

• 6 Vault servers, deployed across three availability zones within a geographic region.
• Using a redundancy zones topology, each node is configured as a voter or non-voter, with one of each per availability zone.
• Locally-attached, high-IO storage for Vault’s integrated storage.
• A layer 4 (TCP) load balancer, with TLS certificates distributed to the Vault nodes.
• An object storage or network filesystem destination for storing automated snapshots.
• An auto-unseal mechanism, such as a cloud-based KMS or an HSM.
• Audit devices, configured to write to multiple destinations.

Using Vault Enterprise Redundancy Zones, the cluster leverages a pair of nodes in each availability zone, with one node configured as a voter and the other as a non-voter. If one voter becomes unavailable — regardless of whether it is an active node or a standby node — autopilot will promote its non-voting partner to take its place as a voter. This architecture gives high availability both within an availability zone as well as across availability zones, ensuring that (n+1)/2 redundancy is maintained at all times.

The resulting failure tolerance: when deployed inline with this architecture, two availability zones can fail with Vault still maintaining quorum, operation, and performance.

The two conditions that warrant DR failover:

1. A rapid loss of all voter nodes, such as failure of all three availability zones.
2. Failure of more than 4 nodes across all three availability zones — a minimum of 2 voter nodes is required to maintain quorum.

Operators should have a runbook for every combination of losing a node or nodes. For example, a node lost in one zone overnight may acceptably wait until the next day for replacement; total loss of connectivity to two zones might trigger immediate replacement, because once 4 nodes are lost the cluster has no more redundancy for maintaining quorum.

Use a layer 4 load balancer capable of handling TCP traffic. Using one reduces static configuration on clients. Any request sent to a standby node that requires an active node to process (e.g. write requests) will be proxied by the standby node to the active node.

Vault server nodes also perform server-to-server communication over 8201/tcp, used for functions critical to cluster integrity such as data replication and active node election. Unlike client requests, server-to-server traffic should not be load-balanced. Instead, each node must be able to communicate directly with all other nodes in the cluster.

Vault automatically enforces encrypted communication between nodes on a separate port. This communication is internal to Vault and does not require additional certificates or management.

TLS — do not manage the TLS certificate for the Vault cluster on the load balancer. Instead distribute the certificate chain onto the Vault nodes themselves, and configure the load balancer to pass through TLS connections to Vault nodes directly. Terminating TLS at the Vault servers offers enhanced security by ensuring a client request remains encrypted end-to-end, reducing the attack surface for potential eavesdropping or data tampering.

Avoid using self-signed certificates for production Vault workloads.

Auto-unseal eliminates the need for manual intervention and reduces the operational complexity of keeping unseal keys secure. Most commonly via cloud KMS — AWS KMS, Azure Key Vault, GCP Cloud KMS.

Then the warning that everyone assumes away:

Recovery keys are purely an authorization mechanism, and cannot decrypt the root key. This means that recovery keys are not sufficient to unseal Vault if the auto unseal mechanism is not working. Using auto-unseal creates a strict Vault lifecycle dependency on the underlying seal mechanism. If the seal mechanism (such as the cloud KMS key) becomes unavailable, or is deleted before the seal is migrated, there is no ability to recover access to the Vault cluster until the seal mechanism becomes available again. If the seal mechanism or its keys are permanently deleted, then the Vault cluster cannot be recovered, even from backups.

Mitigations:

• Careful controls around management of the seal mechanism, such as AWS Service Control Policies.
• If your architecture includes any secondary Vault clusters, configure these clusters with a different seal mechanism than that of the primary. This ensures that if the primary cluster becomes unrecoverable due to an issue with its seal mechanism, the secondary cluster can still be accessed.
• Regardless of the seal mechanism, encrypt the key shares and initial root token value at initialization time with user-supplied public keys, generated from OpenPGP-compliant software such as GPG.

HSM auto-unseal is primarily recommended for private datacenter environments where cloud-based KMS services are not available or not suitable. Vault uses HSM devices supporting PKCS11 to encrypt the root key; Vault stores this HSM-encrypted root key in internal storage, and the HSM device automatically decrypts it.

In addition to the unseal key managed by the HSM, Vault also generates recovery keys to authorize human recovery operations. Because a quorum of recovery key shares can be used to generate a new root token, and because root tokens grant extensive access, PGP encryption should always be used to protect the returned recovery key shares.

Both rekeying the root key and rotation of the underlying data encryption key are supported when using an HSM.

Automated snapshots should be saved to a storage medium separate from the storage used for the Vault storage backend. Snapshot storage does not require the same high IO-throughput requirements as the rest of Vault, but should be durable enough to ensure fast retrieval and restoration. Use object storage such as AWS S3, or a network-mounted filesystem such as NFS.

DR secondary clusters are essentially warm standbys of the primary, used to provide continuity of service in the event of a cluster-wide outage. They do not service any of their own requests, but instead replicate data from the primary. The replicated data not only includes secrets, but also tokens and leases.

If the primary becomes unavailable, operators can manually fail over by promoting the secondary. This restores access to the Vault service and its secrets data, so that clients can reconnect without having to re-authenticate.

• DR secondaries share the same fundamental architecture as primaries.
• They should always be equally sized in order to accept the same volume of requests as the primary.
• Typically placed in different geographical regions to guard against region-level outages, though not required. There must be network connectivity between regions for continuous data replication.
• Automated snapshots cannot be configured for DR secondary clusters before promotion. Manual snapshots only.
• The DR secondary should use auto-unseal, encrypted with a KMS key different from the one used on the primary. After replication is enabled, the primary’s keyring — which contains the recovery keys — is copied to the DR secondary as part of the bootstrapping process. If the secondary also uses KMS auto-unseal, it utilizes the primary cluster’s recovery keys and configuration (e.g. number of fragments and key share threshold). This can be changed with manual configuration.

Failover is manual by design. Promoting and demoting Vault clusters is a non-trivial operation. If Vault briefly encountered service unavailability and the failover process was automated, this could cause unintentional data loss and/or data consistency issues.

Vault operates in an active-passive configuration, meaning only one cluster is the source of truth for all shared secrets in an environment. This mitigates the split-brain problem in which multiple Vault clusters could separately manage the same secrets. The demoted cluster does not attempt to immediately reconnect to a DR primary, but maintains its current dataset so operators can reconnect and update it later.

Clients should not need to reconfigure how they communicate with Vault. Use an additional routing mechanism that can route clients to the active primary cluster regardless of region — Consul, Route53 DNS Failover, Azure Traffic Manager, Google Cloud DNS with HTTP(S) Load Balancer, or F5 GSLB.