Unit 4 — Namespaces and Organizational Design

Namespaces are a method by which a single Vault cluster can be divided into multiple sub-clusters and managed individually. Each namespace can be assigned different login paths and support creating and managing data isolated to their namespace.

Two designed use cases:

• Tenant isolation — a need for isolation between users in policies, secrets, and identities, typically as a result of compliance regulations such as GDPR or internal security policy.
• Self-management — a need to provide delegated administrative privileges for a team to author their own policies and manage their own namespace.

A cluster begins with a root namespace. In self-managed Vault Enterprise clusters, the root namespace should be reserved for Vault operators only — this secures cluster-level configurations, since some configurations can only be made within the root namespace.

Under root, HCP Vault Dedicated automatically creates an admin namespace. Operators of self-managed clusters should create the same namespace and chroot it.

Namespaces can be nested like a folder structure. Each namespace can store secret engines, auth methods, and policies.

Deploy tenant namespaces under the admin namespace. This is where tenant administrators define the policies necessary for granting access to the different sub-namespaces which could map applications or environments in the tenant. The rationale is to consolidate all human access points into a single, centralized location within the cluster. Typically organizations use an identity platform like Active Directory or Okta for human access. Should use cases for mapping auth methods for human users to tenant namespaces subsequently arise, the Vault operator can develop the solution in a controlled manner.

The resulting access model: all users of various business units will authenticate via a single top level auth method. Their membership to groups will dictate which namespace and which secret engine paths within the namespace(s) they will have access to.

Namespaces are subject to limits and maximums within Vault’s backend storage. The effective storage limit on the number of Vault namespaces is a result of the fact that each namespace must have at least two secret engine mounts (for sys and identity), one local secret engine (cubbyhole) and one auth engine mount (token). Depending on your organization’s namespace structure the effective storage maximum will vary.

Antipatterns:

• Not clearly defining criteria for a namespace — created ad-hoc without a clear plan, the structure will quickly become difficult to manage.
• Strong misalignment to organizational structure — namespaces should ideally be roughly aligned to the level of granularity across lines of businesses, divisions, teams, services, apps that needs to be reflected in Vault’s end-state design.
• Non-scalable model or over-segmentation — if an organization plans on provisioning a namespace for each application, but the organization has 10,000 applications to onboard, then Vault’s storage limits will be hit and management of a large number of namespaces will be difficult.

Performance impacts:

• If the number of leases is also growing linearly or exponentially with the number of namespaces and mounts, then API requests might become slower if the growth is unbounded.
• A large list of namespaces can impact the time to complete a leader election. In order for a node to become an active leader in a Vault cluster, it has to load up all the namespaces and mounts. A large number of namespaces will potentially impact the amount of time it takes to complete this process.

That second one converts a design preference into an availability characteristic. It’s the argument that stops namespace-per-application.

Use Terraform to automate the configuration of Vault — namespaces, secret mounts, policies. On a small scale, manually configuring Vault may be manageable. However, as the scale and complexity of your use case increases, you will need some form of automation to help keep up with the increasing management overhead. The example: mapping many OIDC provider groups to Vault external groups across multiple environments can quickly become a significant operational overhead without an automated process.

The Terraform provider is recommended over alternatives such as Ansible because it features testing and support from HashiCorp as well as regular updates to support new features in Vault, and has robust state management which tracks the current configuration on your Vault clusters.

Repository structure:

• A vault-admin-config repository holding cluster configuration as Terraform, mapping to multiple Terraform Enterprise or HCP Terraform workspaces using the VCS workflow — one workspace per environment. Merged PRs trigger apply.
• A consumer repository per application team, initiated by an onboarding request form through ServiceNow, Jira, or equivalent, each mapping to environment-specific workspaces. Pull requests are reviewed and approved by the Vault administration team.

Vault administrators are responsible for: cluster level configuration such as setting up audit logs and mounting auth methods and secret engines; creating common Terraform modules for application owners to consume; approving pull requests in consumer repositories.

Each module should align to a pre-approved Vault use case and provision the resources necessary for consumers to access and work with Vault. For example, a module for consuming Vault static secrets would provision the appropriate Vault policies, identity groups, auth method roles, and the required paths within the KV secret engine. These modules serve as an interface for Vault, abstracting its configuration details from application owners. They should offer sensible defaults and expose only the necessary input parameters.

Application owners should use the Vault UI, CLI, or API to manage secrets to ensure that secrets are not committed into the repository.

Aim to fully shift your onboarding process left, so that the application teams adopting the platform have an end-to-end, automated workflow that is integrated into the organisation’s identity and access management system. This will ensure that the application teams can onboard themselves to the platform without needing to involve the platform team. To not do this risks the situation where manual tasks to onboard internal teams pile up in the platform team’s backlog, this team then becoming a bottleneck to the projects success.

Ensure your project manager has scoped time in the project plan to ensure platform team engineers can complete the necessary engineering so that it can be tested meaningfully in UAT by your early-adopters.

That last sentence is the one to quote in a steering meeting. It converts “we’ll automate onboarding later” into a named project risk.

Separate from production, deploy an additional cluster as a testing instance. A test Vault instance should be able to withstand longer periods of downtime without business impacts, allowing for the safe examination and troubleshooting of new configurations, policies, and updates without putting your business at risk. The configuration of the testing environment should mirror the configuration of the production environment as closely as possible.

Whether to add more depends on the customer. One organization may have CI/CD pipelines for a development environment which are still critical to producing and deploying new software, and therefore may need to be associated with the production Vault cluster to guarantee high-availability and security. Another organization may have extremely strict security requirements for secrets associated with their production environment, and therefore may decide to run a separate Vault cluster for their development environment to prevent any co-mingling of production and development secrets.