Unit 8B — Data Protection: Transit, Transform, Tokenization

Not all personally identifiable information lives in Vault. For example, you may need to store credit card numbers, patient IDs, or social security numbers in external databases. It can be difficult to secure sensitive data outside Vault and balance the need for applications to access the data efficiently while adhering to stringent security standards and protocols.

Vault offers three answers: transit encryption, tokenization, and transforms.

The Vault transit plugin encrypts data, returns the resulting ciphertext, and manages the associated encryption keys. Your application only ever persists the ciphertext and never deals directly with the encryption key.

Vault does NOT store any data encrypted via the transit/encrypt endpoint. The application stores the ciphertext wherever it likes.

vault secrets enable transit

Consumer policy:

path "transit/encrypt/<key_name>" {
  capabilities = [ "update" ]
}

path "transit/decrypt/<key_name>" {
  capabilities = [ "update" ]
}

The tradeoff with encryption is that the ciphertext is often much longer than the original data. Additionally, the ciphertext can contain characters that are not allowed in the original data, which may cause your system to reject the encrypted data. As a result, you may modify your database schema or adjust your data validation system to avoid rejecting the encrypted data.

That schema-change requirement is the whole reason Transform exists. When a customer says “we can’t change the column type,” you’ve found the Transform use case.

Also relevant from Unit 1: 2.0.0 added envelope encryption via Transit — using Transit for in-place encryption workflows where Vault protects data encryption keys and applications encrypt and decrypt local data.

The Vault transform plugin requires a Vault Enterprise license with Advanced Data Protection (ADP). Introduced in Vault Enterprise 1.4 with the Advanced Data Protection module.

Three transformations: format preserving encryption, data masking, and tokenization.

Format preserving encryption

FPE is a two-way transformation that encrypts external data while maintaining the original format and length. For example, transforming a credit card number to an encoded ciphertext made of 16 numbers.

Unlike stand-alone encryption, FPE maintains the original length and data structure for encoded data so the transformed data works with your existing database schema and validation systems. And, unlike tokenization, FPE preserves the original structure without the risk of leaky abstraction.

Vault uses the FF3-1 algorithm — NIST vets, updates, and tests the FF3-1 algorithm to protect against specific types of attacks and potential future threats from supercomputers. That NIST provenance is the sentence to use with a compliance team.

FPE transformation is stateless, which means that Vault does not store the protected secret. Instead, Vault protects the encryption key needed to decrypt the ciphertext. By only storing the information needed to decrypt the ciphertext, Vault provides maximum performance for encoding and decoding operations while also minimizing exposure risk for the data.

Built-in transformation templates exist for common data like credit card numbers and US social security numbers, and custom transformation templates are supported — you can use regular expressions to specify the values you want to transform and enforce a schema on the encoded value.

One caveat to raise with customers: after transformation and formatting by the template, the value is an encrypted version of the input with the format preserved. However, the value itself may be invalid with respect to other standards. For example the output credit card number may not validate (it likely won’t create a valid check digit). So one must consider when the outputs are stored whether validation in storage may reject them.

Format-preserving is not validation-preserving. That distinction catches people.

Data masking

A one-way transformation that replaces characters on the input value with a predefined translation character.

Masking is non-reversible. Not recommended for situations where you need to retrieve and decode the original value.

Good when you need to show or print sensitive data without full readability. For example, masking a bank account number in an online banking portal to prevent potential security breaches from bad actors who might be observing the screen.

Tokenization

Tokenization exchanges a sensitive value for an unrelated value called a token. The original sensitive value cannot be recovered from a token alone — they are irreversible. Unlike format preserving encryption, tokenization is stateful. To decode the original value, the token must be submitted to Vault where it is retrieved from a cryptographic mapping in storage.

Unlike standalone tokenization, tokenization with Vault transform is a two-way, random encoding that satisfies the PCI-DSS requirement for data irreversibility while still allowing you to decode tokens back to their original plaintext.

That sentence is the pitch. It resolves what looks like a contradiction in a PCI conversation: irreversible to an attacker, decodable by an authorized caller.

Cryptosystem: AES256-GCM96 for encryption of its token store, with keys derived from the token and a tokenization root key.

Vault transform creates a new key for each tokenization transformation, which helps ensure a strong cryptographic distinction between different tokenization use cases. For example, a credit card processor may want to distinguish between the same credit card number used by different merchants without having to decode the token.

Convergent tokenization. By default, tokenization produces a unique token for every encode operation. Vault transform supports convergent tokenization, which lets you use the same encoded value for a given input. This enables statistical analysis of the tokens in your system without decoding the token — for example, counting the entries for a given token, querying relationships between a token and other fields in your database, and relating information that is tokenized across multiple systems.

The cost: convergent tokenization has a small performance penalty in external stores and a larger one in the built in store due to the need to avoid duplicate entries and to update metadata when convergently encoding. It is recommended that if one has some use cases that require convergence and some that do not, one should create two different tokenization transforms with convergence enabled on only one.

Token lookup — looking up a token given its plaintext — is ordinarily contrary to the nature of tokenization where we want to prevent the ability of an attacker to determine that a token corresponds to a plaintext value (a known plaintext attack). It is supported, but only in some configurations: when convergence is enabled, or if the mapping mode is exportable and the storage backend is external.

The Tokenization transform is designed to resist a number of attacks on the values produced during encode. In particular it is designed so that attackers cannot recover plaintext even if they steal the tokenization values from Vault itself. In the default mapping mode, even stealing the underlying transform key does not allow them to recover the plaintext without also possessing the encoded token. An attacker must have gotten access to all values in the construct.

In the exportable mapping mode however, the plaintext values are encrypted in a way that can be decrypted within Vault. If the attacker possesses the transform key and the tokenization mapping values, the plaintext can be recovered. This mode is available for the case where operators prioritize the ability to export all of the plaintext values in an emergency, via the export-decoded operation.

exportable is a deliberate downgrade of the security property. HashiCorp’s guidance is explicit: the exportable mode is only recommended if this use case is required, as the default cannot be decoded by attackers even if they gain access to Vault’s storage and keys.

Make a customer choosing exportable say out loud what emergency they’re protecting against. It is frequently chosen by default without anyone understanding what was traded away.

Metadata: since tokenization isn’t format preserving and requires storage, one can associate arbitrary metadata with a token. Metadata is considered less sensitive than the original plaintext value. As it has its own retrieval endpoint, operators can configure policies that may allow access to the metadata of a token but not its decoded value to enable workflows that operate just on the metadata.

That’s a useful least-privilege pattern — analytics teams get metadata, never plaintext.

This is where an RA earns their keep, because the three options scale completely differently.

Tokenization transformation is stateful, which means the encode operation must perform writes to storage on the primary node of your Vault cluster. As a result, any storage performance limits on the primary node also limits scalability of the encode operation.

In comparison, neither Vault transit encryption nor FPE transformation write to storage, and both can be horizontally scaled using performance standby nodes.

Detail on the internal store: this differs from some secret engines in that the encode and decode operations require an access of storage per operation. Other engines use storage for configuration but can process operations largely without accessing any storage. All other operations can be performed on secondaries.

And a consistency property to design around: due to replication, writes to the primary may take some time to reach secondaries, so other read operations like decode or metadata may not succeed on the secondaries until this happens. In other words, tokenization is eventually consistent.

An application that encodes then immediately decodes against a different cluster will fail intermittently. That is a design constraint, not a bug.

External storage: for high-performance use cases, configure Vault to store the token mapping in an external database. External stores can achieve a much higher performance scale and reduce the load on the internal storage for your Vault installation. All nodes (except DRs) can participate in all operations using external storage, but one must take care to monitor and scale the external storage for the level of traffic experienced. The storage schema is simple however and well known approaches should be effective.

Supported external stores: PostgreSQL, MySQL, and MSSQL. The Schema Endpoint may be used to initialize and upgrade the necessary database tables. Vault uses a schema versioning table to determine if it needs to create or modify the tables. If you make changes to those tables yourself, the automatic schema management may become out of sync and may fail in the future.

External stores may often be preferred due to their ability to achieve a much higher scale of performance, especially when used with batch operations.

Cross-reference Unit 10.2: this is another consumer of the CPU guidance — CPU monitoring is particularly important if the Transit or the Transform secrets engines are in use, because encryption can place a heavy demand on the CPU.

TTLs. By default, tokens are long lived, and the storage for them will be maintained indefinitely. Where there is a concept of time-to-live, it is strongly encouraged that the tokens be generated with a TTL. For example, as credit cards have an expiration date, it is recommended that tokenizing a credit card primary account number be done with a TTL that corresponds to the time after which the PAN is invalid.

This allows such values to be tidied and removed from storage once expired. Tokens themselves encode the expiration time, so decode and other operations can immediately reject the operation when presented with an expired token.

Without TTLs the token store grows forever. Same discipline as PKI tidying in Unit 8.6 and lease hygiene in Unit 7.3.

Key management. Tokenization supports key rotation. Keys are tied to transforms, so key names are the same as the name of the corresponding tokenization transform. Keys can be rotated to a new version, with backward compatibility for decoding. Encoding is always performed with the newest key version. Key versions can be tidied as well. Keys may also be rotated automatically on a user-defined time interval, specified by the auto_rotate_field of the key config.

From the Transform overview: each configured tokenization transformation keeps a set of versioned keys. When a key rotates, older key versions, within the configured age limit, are still available for decoding tokens generated in the past. Vault cannot decode generated tokens with keys below the minimum key version.

Set the minimum key version too aggressively and you permanently orphan old tokens. Treat that setting with the same care as a certificate revocation.

Snapshot and restore. Snapshot allows one to iteratively retrieve the tokenization state, for backup or migration purposes. The resulting data can be fed to the restore endpoint of the same or a different tokenization store. Note that the state is only useable by the tokenization transform that created it, as state is encrypted via keys in that configured transform.

Zero-downtime store migration. Tokenization stores are configured separately from the tokenization transform, and the transform can point to multiple stores. The primary use case for this one-to-many relationship is to facilitate migration between two tokenization stores. When multiple stores are configured, Vault writes new tokenization state to all configured stores, and reads from each store in the order they were configured.

The documented procedure:

1. Configure the new tokenization store in the API.
2. Modify the existing tokenization transform to use both the existing and new store.
3. Snapshot the old store.
4. Restore the snapshot to the new store.
5. Perform any desired validations.
6. Modify the tokenization transform to use only the new store.

Worth memorizing — “how do we move off the internal store once we outgrow it?” is the predictable follow-up to any tokenization deployment, and this is a clean answer.

Two questions that resolve most of this in discovery: can you change the database schema? and do you ever need the original value back?

Prerequisites before deploying the Transform engine:

• Vault Enterprise Deployment: the Transform engine is a feature of Vault Enterprise and is available with the ADP license.
• Authentication and Authorization Configuration: proper authentication methods and role-based access controls should be configured to secure access to the Transit and Transform engines.
• Data Classification and Compliance Assessment: before implementing the Transform engine, conduct a thorough assessment of the data types that require protection and ensure that the configuration aligns with the relevant regulatory compliance requirements, such as GDPR, PCI-DSS, or HIPAA.

That third one is the RA’s job, and it comes before configuration. Which transformation you choose falls out of data classification — you cannot pick FPE vs tokenization vs masking without knowing what the data is and which regulation applies.

Scaling considerations:

• Read-oriented operations: the Transit engine and most Transform engine operations (excluding tokenization) are primarily read-oriented. This allows these operations to benefit significantly from horizontal scaling. Adding non-voter (performance standby) nodes to your Vault cluster increases capacity to handle Transit and Transform operations, as these nodes can serve read requests without participating in the consensus protocol.
• Tokenization: involves high-frequency write operations to store plaintext-token mappings. This can place substantial load on Vault’s storage backend. To optimize performance, use external SQL stores — MySQL, PostgreSQL, or Microsoft SQL Server — to offload write-intensive tokenization tasks from Vault’s internal storage to an external database, leveraging the database’s ability to handle high write loads efficiently, while allowing Vault to scale its other operations effectively.

Note how this maps onto the redundancy zone architecture in Unit 2.3: the non-voter in each AZ is the capacity you’re adding for Transit and FPE throughput.

HVD’s four-step scaling strategy:

1. Monitor performance: regularly track key performance indicators such as CPU utilization, throughput, and latency. Establish alerts based on predefined thresholds to proactively scale your deployments before they impact performance.
2. Optimize load balancing: configure your load balancers to distribute read traffic efficiently across non-voter nodes. This approach reserves voter nodes for write operations and cluster management tasks.
3. Scale CPU and memory: since Transit and Transform operations are CPU-intensive due to cryptographic processing, scaling up CPU resources on Vault nodes can improve throughput and reduce latency. Ensure sufficient memory is available to handle concurrent operations effectively.
4. Add performance replicas: for additional scaling, consider adding performance replicas. This reduces the load on the primary cluster and improves response times.

Step 2 is the same load balancer health-check configuration as Lab H.2 and the PKI no_store=true discussion in Unit 8.5 — perfstandbyok=true is what lets read traffic reach the non-voters. One configuration, three capabilities depending on it.

For the consumer-side workflow, see the HVD User Guide pages on Transit and Transform secrets engines and their worked examples.