Unit 6 — Machine Authentication

In order for a client to interact with Vault’s API, it must pass an authentication token. For a human user, providing a proof of identity is straightforward using a user ID and password. However, this is more challenging for applications because you need a secure way to provide that initial credential. This problem is often referred to as the secret zero or secure introduction challenge. How does a secret consumer prove that it is the legitimate recipient for a secret so that it can acquire a token? How can you avoid persisting raw token values during your secure introduction?

Two approaches: platform integration and trusted orchestrator.

Vault trusts the underlying platform (e.g. AWS, Azure, GCP, Kubernetes etc.) which assigns a token or cryptographic identity (such as IAM token, signed JWT) to virtual machine, container, or serverless function. Vault uses the provided identifier to verify the identity of the client by interacting with the underlying platform. After the client identity is verified, Vault returns a token to the client that is bound to their identity and policies.

Worked example: an application on an AWS EC2 instance. When that instance is started, an IAM token is provided via the machine local metadata URL. That IAM token is provided to Vault as part of the AWS auth method. Vault uses that token to query the AWS API and verify the token validity and fetch additional metadata about the instance (account ID, VPC ID, AMI, region). These properties are used to determine the identity of the client and to distinguish between different roles (e.g. a web server versus an API server).

When the client app is running on a compute construct hosted on a supported cloud provider, you can leverage the corresponding auth method.

You have an orchestrator which is already authenticated against Vault with privileged permissions. The orchestrator launches new applications and injects a mechanism they can use to authenticate (e.g. PKI cert, token, AppRole etc.) with Vault.

Worked example: Terraform as a trusted orchestrator. Terraform already has a Vault token with enough capabilities to generate new tokens or create new credentials. Terraform can interact with platforms such as VMware to provision new virtual machines. VMware does not provide a cryptographic identity, so a platform integration is not possible. Instead, Terraform can create and inject an AppRole credential into a new machine, acting as a trusted orchestrator that extends trust to the machine.

Vault Agent is a client daemon which automates the workflow of client login and token refresh. It can be used with either platform integration or trusted orchestrator approaches.

The decision is one question: does the platform this workload runs on issue it a cryptographic identity? Yes → platform integration. No → trusted orchestrator. There is no third option that isn’t a static secret.

The aws auth method provides two ways to authenticate using either iam principals or ec2 instances. The ec2 method was implemented before the primitives to implement the iam method were supported by AWS. The iam method is the recommended approach as it is more flexible and aligns with best practices to perform access control and authentication.

With the iam method, an authenticating client creates a signed GetCallerIdentity AWS request and uses it to authenticate. The request is signed using the IAM credentials that are automatically supplied to AWS instances in IAM instance profiles, Lambda functions, and others. When Vault receives the login request, it can execute the STS query without needing to know its contents. Vault then authenticates the client by comparing the IAM principal’s ARN mapped to the Vault role. Clients must have an ARN that matches one of the ARNs bound to the role.

vault auth enable aws

vault write auth/aws/config/client \
    secret_key=<secret> \
    access_key=<key>

vault write auth/aws/config/client \
    iam_server_id_header_value=vault.example.com

vault write auth/aws/role/demo auth_type=iam \
    bound_iam_principal_arn=arn:aws:iam::123456789012:role/MyRole \
    policies=default \
    token_ttl=8h \
    max_ttl=8h

The replay-attack control worth recommending unprompted: require an additional header, X-Vault-AWS-IAM-Server-ID. Clients must include this header, and it must be among the signed headers validated by AWS. This protects against different types of replay attacks, for example a signed request sent to a dev server being resent to a production server. This header should be set to the Vault server’s DNS name.

Permissions: the IAM account or role must allow iam:GetUser and iam:GetRole. If Vault is deployed on AWS EC2 instances, Vault will attempt to use standard environment variables or IAM EC2 instance role credentials without having to embed credentials into the auth method configuration.

Wildcards are supported at the end of the ARN — arn:aws:iam::123456789012:* matches any IAM principal in that account.

HVD’s positioning, quoted: “While AppRole is a valid use case, use this auth method as a last resort. Relying on alternative platform authentication methods is inherently more secure because AppRole can be misconfigured, leading to a less secure posture than is possible.”

AppRole is not a trusted third-party authenticator, but a trusted broker method. The onus of trust rests in a securely-managed broker system that brokers authentication between clients and Vault. Three players: Vault, the broker, and the secret consumer.

Multiple SecretIDs can be created for the same RoleID. The value of each SecretID for the same RoleID is unique and they can all be used to authenticate to Vault. For this reason, it is important to set a TTL for each SecretID generated to prevent the risk of a leaked SecretID.

vault auth enable approle          # -path=apps to mount elsewhere

vault write auth/approle/role/demo \
  secret_id_num_uses=1 \
  secret_id_ttl=15m \
  token_policies="default" \
  token_ttl=1h \
  token_max_ttl=1h

secret_id_num_uses determines how many times a particular SecretID can be used to authenticate. Setting this value to 1 effectively makes any generated SecretID a one time password. The default value is 0, which will allow unlimited uses. secret_id_ttl ensures that any generated SecretIDs that did not get used expire automatically. Always set these two settings to reduce the attack surface in case the SecretID was accidentally leaked.

Consequence to design for: limiting the SecretID TTL and its number of uses means that a new SecretID will need to be delivered to the application once the application’s Vault token reaches its maximum TTL and can no longer be renewed.

Full parameter example:

vault write auth/approle/role/jenkins \
      secret_id_bound_cidrs="0.0.0.0/0","127.0.0.1/32" \
      secret_id_ttl=60m \
      secret_id_num_uses=5 \
      enable_local_secret_ids=false \
      token_bound_cidrs="0.0.0.0/0","127.0.0.1/32" \
      token_num_uses=10 \
      token_ttl=1h \
      token_max_ttl=3h \
      token_type=default \
      period="" \
      policies="default","test"

The delivery model

A potential risk with the trusted orchestrator pattern is that if the trusted orchestrator is responsible for delivering the AppRole credentials, it would also have access to the applications’ secrets. If the trusted orchestrator is compromised, then application secrets would be at risk as well.

To prevent any one system, other than the target client, from obtaining the complete set of credentials, deliver those values separately through two different channels. This enables you to provide narrowly-scoped tokens to each trusted orchestrator to access either RoleID or SecretID, but never both.

The central tenet: during the brokering of the authentication to Vault, the RoleID and SecretID are only ever together on the end-user system that needs to consume the secret.

RoleID is not a secret value. It’s a static UUID that identifies a specific role configuration. Because it is not a secret, you can embed the RoleID value into a machine image or container as a text file or environment variable — built with Packer, provisioned with Terraform. Generally, you create a role per application to ensure that each application will have a unique RoleID.

path "auth/approle/role/jenkins/role-id" {
   capabilities = [ "read" ]
}

SecretID requires binding CIDRs and response wrapping.

Response wrapping guarantees confidentiality, integrity, and non-repudiation of SecretID. Instead of providing the SecretID in plaintext, Vault puts it into a new token’s Cubbyhole with a token use count of 1. When the application attempts to read the SecretID, we can guarantee that only this application can read it.

vault write -wrap-ttl=60s -force auth/approle/role/jenkins/secret-id

path "auth/approle/role/jenkins/secret-id" {
   capabilities = [ "update" ]
}

Make wrapping mandatory by policy:

# This effectively makes response wrapping mandatory for this path by setting min_wrapping_ttl to 1 second.
# This also sets this path's wrapped response maximum allowed TTL to 90 seconds.
path "auth/approle/role/my-role/secret-id" {
    capabilities = ["create", "update"]
    min_wrapping_ttl = "1s"
    max_wrapping_ttl = "90s"
}

Delivery ordering

Deliver RoleID first, let the application start, then generate and deliver the SecretID. Generating it earlier is sometimes done, however this order is the most secure way to deliver the SecretID to ensure that the application is fully started before the SecretID is delivered for immediate use. This minimizes the risk of exposure of an unconsumed SecretID.

Ideally, application instances should wait for the SecretID and consume it immediately, or they should terminate and restart (e.g. by a supervising orchestrator) until the SecretID is available. In some cases, pre-delivery may be necessary if the application cannot gracefully handle startup before the SecretID is present.

Ensure the wrapping token is delivered through a different channel than the RoleID. Promptly remove the wrapping token once it is used — if you are using the Vault agent, it will attempt to delete the SecretID file by default after it has been read, and automatically unwraps the SecretID when a wrapping token is detected.

Delaying its use increases the risk of an exposed credential, expiration of the wrapping token or SecretID, and the potential for a compromised token to go undetected until authentication is attempted.

Monitoring tripwires

• Configure your CI/CD pipeline to raise a security alert to the Platform Team should this method result in a 500 error being received by an application attempting to use the wrapping token. Wrapping tokens are single use only, and a 500 API response code from Vault indicates that this token is now invalid suggesting it has already been used. Test this alerting mechanism to ensure it works as expected.
• If Vault throws a use-limit error when an application tries to read the SecretID, you know that someone else has read the SecretID. The audit logs will indicate where the read attempt originated.

Role design

Take the time to consider role management before enabling the auth method. If you intend to delegate role management to others, assess whether they require full access to all roles. If not, it might be beneficial to enable AppRole at multiple paths, enabling targeted policies for managing specific sets of roles.

Follow the principle of least privilege for each application — if two applications use the same role, they will have access to the same secrets. Roles can have multiple attached policies. To handle shared and unique secrets among apps, craft one policy for common secrets and app-specific policies for unique needs.

The three anti-patterns

1. CI worker retrieves secrets and passes them to the runner. The worker would have to have the authorization to retrieve many secrets, none of which is the consumer. Additionally, if a single secret were to become compromised, then there would be no way to tie an identity to it and initiate break-glass procedures on that identity. So all secrets would have to be considered compromised.
2. CI worker passes RoleID and SecretID to the runner. While this prevents the worker from having Vault’s authorization to retrieve all secrets, it has that capability as it has both RoleID and SecretID.
3. CI worker passes a Vault token to the runner. The worker will have access to the child tokens that would have authorization.

Jenkins pattern: Jenkins itself will need an identity; however, you should never have Jenkins log into Vault and pass a client token to the application via workflow. Jenkins needs to give the application its own identity so that the application gets its own secret. Obscure the RoleID from Jenkins but allow Jenkins to deliver a wrapped SecretID.

path "auth/approle/role/+/secret*" {
  capabilities = [ "create", "read", "update" ]
  min_wrapping_ttl = "100s"
  max_wrapping_ttl = "300s"
}

Because the worker’s token is limited to retrieving wrapped SecretIDs, the worker could be pre-seeded with a long-lived Vault token or use a hard-coded RoleID and SecretID as this would present only a minor risk.

In any trusted broker situation, the broker must be secured and treated as a critical system. This means that users should have minimal access to it and the access should be closely monitored and audited.

In general, with any auth method, it’s preferable for applications to keep using the same Vault token to fetch secrets repeatedly instead of a new authentication each time. Authentication is an expensive operation and results in a token that Vault must keep track of. If high authentication throughput, 1000s of authentications per second, are expected we recommend using batch tokens which are issued from memory and do not consume storage.

A long token TTL can cause out of memory when trying to purge millions of AppRole leases. To avoid this, reduce TTLs for AppRole tokens and implement token renewal where possible. You can increase the memory on the Vault server; however, it won’t be a long-term solution.

Consider running Vault Agent on the client host, and let the agent manage the token’s lifecycle. Vault Agent reduces the number of tokens used by the client applications. In addition, it eliminates the need to implement the Vault APIs to authenticate with Vault and renew the token TTL.

Identity brokering occurs when Vault translates a workload’s native platform identity into temporary access for another system without creating new static secrets.

True brokering: a Kubernetes pod authenticates with its service account and Vault generates temporary AWS credentials for S3; an EC2 instance authenticates with its IAM role and Vault creates a short-lived PostgreSQL user.

Not brokering:

• CI/CD secret injection — a pipeline authenticates, fetches secrets, and injects them as environment variables. This is not identity brokering because the pipeline uses its identity, not the application’s runtime identity, to obtain the secrets. While this does provide automation and potentially qualifies as an evolutionary improvement of the legacy secrets-focused approach, it does not achieve zero trust.
• Using long-lived AppRoles — the AppRole is a new set of credentials, not the workload’s native platform identity. This replaces one static secret with another.

Most customers who believe they’ve “adopted Vault” are doing the first. Name it precisely without dismissing it — it’s a productive way to open a maturity conversation.

Useful counter to a common objection: it is a common misconception that dynamic secret patterns give you less insight into machine access across your environment. On the contrary, brokered identity transactions on Vault are deeply auditable, much more so than static credentials and their inevitable sprawl.

NIST alignment. The model implements SP 800-207 tenets directly: no implicit trust granted based solely on physical or network location; access granted on a per-session basis via leases and short-lived credentials; access determined by dynamic policy including observable client identity; and all resource authentication and authorization dynamic and strictly enforced before access is allowed. For customers whose security architecture is written against 800-207, this maps Vault onto language their auditors accept.