Domain 1: Create a Working Vault Server Configuration
- The storage stanza for Raft requires the path parameter, the filesystem location where Vault writes the Raft log store (raft.db) and FSM snapshots; without it Vault has nowhere to persist cluster state and cannot start.
- If node_id is left unset, Vault generates a random persistent UUID at first startup and writes it to a node-id file in the Raft data directory so identity survives restarts; operators often set it explicitly (for example to a cloud instance ID) for easier troubleshooting.
- api_addr and cluster_addr must both be set correctly: cluster_addr carries Raft's own peer-to-peer transport while api_addr is what Vault uses to redirect clients to the active node; getting either wrong commonly breaks joining, replication, or client redirects even though the process appears to start fine.
- retry_join lets a server automatically join an existing Raft cluster at startup by contacting one of several listed peer addresses, removing the need to run vault operator raft join by hand; performance_multiplier scales Raft's internal timing (heartbeat interval, election timeout) so a cluster tolerates higher-latency networks without spurious elections, defaulting to 5 on Community Edition and 1 on Enterprise.
- Integrated Storage bundles Raft consensus directly into the Vault binary, eliminating the need for a separate Consul cluster to install, secure, and upgrade; Consul remains supported, but Raft is the current recommended default. The file backend only supports a single active server (no HA) and inmem is not durable at all, so neither suits production.
- By default Vault seals with Shamir's Secret Sharing: vault operator init returns a configurable number of key shares (for example 5) and a threshold (for example 3) of them must be supplied to vault operator unseal to reconstruct the master key before Vault will decrypt anything.
- Auto-unseal delegates the unseal step to a cloud KMS (AWS KMS, Azure Key Vault, GCP Cloud KMS) or to Transit, storing a wrapped root key so a restarted node unseals itself automatically - essential for HA clusters and automated recovery without operators keying in shares.
- vault operator rekey generates a brand-new set of Shamir unseal keys (and can change the share count or threshold) without changing the underlying encryption key; vault operator rotate rotates the underlying encryption key itself, keeping older key versions available to decrypt data written before the rotation.
- vault operator init bootstraps the very first Raft node as a single-member cluster and its initial leader; a correctly configured retry_join then lets subsequent nodes join automatically the moment they unseal, or vault operator raft join can trigger the join manually if retry_join failed at boot.
- vault operator raft list-peers reports each server's Raft ID, address, and voter or leader status, and remove-peer permanently drops a dead server from the configuration. If the voting majority is lost, recovery uses a peers.json file to force a new peer set, or vault operator raft snapshot save, restore, and inspect to recover from (or verify) a known-good backup; Autopilot's cleanup_dead_servers (default false) and min_quorum control how aggressively dead servers get auto-pruned.
Domain 2: Monitor a Vault Environment
- Vault ships three built-in audit device types - file, syslog, and socket - each writing structured, tamper-evident JSON entries to a different destination: a local file, the system syslog daemon, or a network endpoint; file_path and mode are specific to the file device.
- vault audit enable with no path= argument mounts the device at the default path matching its type (for example file/); syslog takes tag and facility so rsyslog or syslog-ng can route entries, and socket takes address and socket_type (tcp, udp, or unix) to stream entries to a network endpoint.
- vault audit list, with -detailed to see each device's configured options such as hmac_accessor, is the command for enumerating every currently enabled audit device, its mount path, and its type.
- vault audit disable only unmounts the device going forward; it has no effect on entries already written to disk, so retention, archival, and deletion of historical logs remain the operator's responsibility.
- Vault blocks a request rather than let it proceed unaudited only when ALL enabled audit devices fail at the same time; it does not reseal itself, it simply denies that one request until at least one device is healthy again. Running multiple independent devices is the standard defense against this becoming an outage.
- Every audit device HMACs sensitive values before writing them, preserving a correlatable, tamper-evident record - identical inputs hash identically within a device - without ever exposing the underlying plaintext secret.
- Each audit device holds its own independent, Vault-generated HMAC salt protected by the storage barrier and never operator-supplied, which is why the same secret value hashes differently across two different audit devices.
- hmac_accessor defaults to true and controls only whether the token accessor field is hashed in audit logs; an accessor is a lookup handle, not a bearer credential, so leaving it visible is useful for grepping and cross-referencing lookups without weakening security.
- The telemetry stanza in the Vault configuration exposes operational metrics - request latency, token and lease counts, storage backend health, Raft state - to a metrics sink such as Prometheus, StatsD, or Circonus; the sys/metrics API endpoint can also be scraped directly.
- Combining audit logging (who did what) with telemetry (how the system is performing) gives operators both a compliance-grade activity trail and the operational visibility needed to catch degraded storage, replication lag, or capacity problems before they become outages.
Domain 3: Employ the Vault Security Model
- ACL capabilities are create, read, update, delete, list, sudo, and deny; create and update together cover writes whether or not data already exists at that path, so granting only those two still returns 403 permission denied on read, list, or delete requests.
- Browsing a KV v2 folder, in the UI or with vault kv list, issues a list request against the metadata path and needs the list capability there, while reading one already-known secret only needs read on the data path - so read can succeed even where list on metadata is missing.
- Root-protected endpoints, such as raw storage access, require the sudo capability in addition to whatever other capability would otherwise apply, even for a non-root token.
- A deny capability on any attached policy always overrides every allow granted by every other attached policy for that path - Vault evaluates the union of all attached policies, but deny wins unconditionally no matter how many other policies allow the request.
- When multiple path blocks within a single policy could match a request, the most specific matching path block governs the decision, not the order the blocks were written in.
- In ACL path syntax, the plus (+) wildcard matches exactly one path segment's worth of characters, while the asterisk (*) wildcard is only valid as a trailing character and matches everything after it.
- Fine-grained request control uses denied_parameters (blacklists a key - including it anywhere in the payload fails the whole request), allowed_parameters (an empty value list acts as a wildcard for that key), required_parameters (the caller must include the named key or the request is denied), and min_wrapping_ttl and max_wrapping_ttl (bound the acceptable response-wrapping TTL header).
- The default policy attaches automatically to most tokens and grants only limited self-service actions like token lookup and renewal; the root policy is a fixed, immutable policy that bypasses ACL evaluation entirely and is held only by root tokens.
- ACL policies can template identity into path rules with no separate declaration required, since Vault auto-detects the templating syntax: identity.entity.metadata.<key> resolves custom entity metadata and identity.groups.names expands to the entity's current group membership names, both evaluated per request against the caller's own entity - the practical distinction from ordinary policy attachment is scope, path-wide versus identity-scoped.
- Enterprise adds two more layers to the security model: namespaces isolate policies, auth methods, and secrets engines into independently administered tenants within one cluster, and Sentinel governance policies add enforcement beyond ACLs - soft-mandatory blocks by default but allows a privileged override, hard-mandatory always blocks with no override path, and advisory only logs failures without ever blocking.
Domain 4: Build Fault-Tolerant Vault Environments
- In a standard HA cluster, only the active node touches the storage backend directly; standby nodes stay unsealed and fully Raft-replicated but transparently forward client requests to the active node over the cluster port and relay its response back, so the client experiences a normal call.
- A standby answers basic status or health checks, such as vault status or /sys/health, locally from in-memory node state without forwarding, since those do not touch the storage backend; anything that reads or writes actual secret data must go through the active node.
- Raft consensus guarantees exactly one active leader at a time; every other voting member is a follower applying the same replicated log, regardless of how gracefully the previous leader left. Quorum is a function of total voter count - a 3-node cluster tolerates 1 failure, a 5-node cluster tolerates 2 - and adding voters is mainly about fault tolerance, not throughput, since writes still funnel through one active node.
- When the active node is lost, the cluster does not seal or require manual rejoining: the surviving voters detect missed heartbeats, hold a Raft election, and, as long as a majority survives, automatically elect one of themselves as the new active node.
- Request forwarding is the default standby behavior - the standby proxies the call and relays the response, so the client sees a normal response with no special handling required. Disabling forwarding switches standbys to returning a redirect instead, trading a proxy hop for a direct client connection that requires client-side redirect support; either way the request still reaches the active node.
- Forwarded requests and Raft consensus messages both travel over the dedicated cluster port (8201), separate from the client-facing API port (8200); blocking or misconfiguring 8201 or the cluster address isolates a node from Raft and forwarding even though its API port may still answer, and cuts it off from leadership eligibility.
- vault operator step-down triggers a planned, graceful leadership handoff useful before a maintenance reboot or controlled failover testing; it only relinquishes leadership on whichever node currently holds it, that node stays sealed-open and running, and the resulting election has no guaranteed winner among the healthy voters.
- Disaster Recovery (DR) replication, an Enterprise feature, continuously ships the entire encrypted dataset from a primary cluster to a standby cluster in another site; the DR secondary is not accessible for normal traffic and exists purely to be promoted if the primary site is lost.
- Performance Replication, also Enterprise, links independent Vault clusters so writes made on the primary asynchronously replicate to one or more performance secondaries, letting each site serve local reads (and local writes for local-only mounts) with lower latency - distinct from DR replication, which is a pure standby rather than an active read target.
- A fault-tolerant production topology typically layers Raft HA within a site for fast, automatic failover via voters, Performance Replication across sites for low-latency local access, and DR Replication to a separate region for full-site disaster recovery - each mechanism covers a different scope of failure.
Domain 5: Harden Vault for Production
- Production listener stanzas must terminate TLS themselves: set tls_disable = 0 (the default) and reference tls_cert_file and tls_key_file so the Vault-to-client hop is encrypted end to end; terminating TLS only at an upstream proxy leaves the proxy-to-Vault hop in cleartext.
- Vault automatically generates and rotates its own internal mutual TLS certificates for node-to-node cluster traffic on port 8201, separate from the listener's client-facing TLS, so Raft and HA replication traffic is protected without any extra configuration.
- tls_min_version on the listener restricts negotiation to modern protocol versions such as TLS 1.2 or 1.3, closing off downgrade attacks against clients that still support older, weaker TLS versions.
- Vault supports reloading TLS certificates without a restart: write the new cert and key files to the configured paths, then send SIGHUP so Vault re-reads the listener configuration, avoiding any client-visible downtime.
- Requiring mutual TLS needs both tls_require_and_verify_client_cert = true and tls_client_ca_file configured together, so Vault can validate presented client certificates against a trusted CA; setting either alone is not sufficient.
- With disable_mlock = false, the default posture to keep, Vault calls mlock to pin its process memory in RAM so the OS cannot page unseal keys or decrypted secrets out to swap space where they could later be recovered from disk; setcap cap_ipc_lock grants a non-root process exactly the capability it needs to do this.
- When mlock truly cannot be used, for example in certain containerized environments, the compensating controls are disabling swap entirely and restricting host access, keeping decrypted secrets in RAM only and never paged to disk even without mlock itself.
- Run Vault under a dedicated, low-privilege service account rather than root, with the config and data directories locked down to that account only, and pin the systemd unit to it with a non-interactive shell, limiting what a compromised Vault process could reach.
- Deploy Vault on a single-tenant, dedicated host with a minimal installed package set, reducing both the blast radius of a colocated compromise and the number of exploitable components on the box.
- Firewall both Vault ports deliberately: 8200, the client API and UI port, should be reachable only from authorized clients rather than the open internet, and 8201, the internal cluster port, should be restricted to peer nodes only; the Raft data directory and any snapshot files should be locked down to the dedicated vault account with the same rigor as the live storage backend, even though the data itself is encrypted at rest.
Domain 6: Scale Vault for Performance
- Vault Enterprise's performance standby feature lets standby nodes answer plain reads locally, from a continuously replayed stream of write-ahead log (WAL) entries sent by the active node, instead of forwarding every request; only requests that write to storage still get forwarded.
- disable_performance_standby = true turns every standby back into a plain HA standby that forwards everything, reads included, trading away the read-scale benefit for uniform, simpler request handling; it is a static, per-node server configuration setting that requires a restart to take effect, not a live API toggle or a SIGHUP reload.
- Performance standby requires a Vault Enterprise license; Community Edition standby nodes always forward every request to the active node regardless of node count, non_voter status, or storage backend.
- vault status exposes a Performance Standby Node field (true or false) stating whether a node is currently serving as a performance standby, and a Performance Standby Last Remote WAL field showing how far its applied state trails the active node's latest index - useful checks before pulling a node for maintenance.
- A performance standby that is not caught up on WAL replication either waits briefly or forwards the request rather than ever serving stale data, so read consistency is preserved even under replication lag.
- Every non-active node in the cluster can act as a performance standby simultaneously - there is no per-cluster cap on how many nodes serve reads this way.
- Dynamic credential generation is forwarded despite being an HTTP GET because it writes a lease to storage; a plain KV v2 read has no storage side effect, so it is exactly the kind of request performance standby is built to serve locally.
- Raft non-voter nodes add read capacity without joining the write-commit quorum: more voting members means more write-replication overhead in exchange for fault tolerance, while non-voters add read scale without that overhead, so for read scale confined to a single site, adding standby-capable nodes to the existing cluster is the simplest fix.
- The read-scale benefit only materializes if client traffic is actually routed to the standby nodes, for example via a load balancer that spreads reads across the cluster; a node coming out of the active role must first resync its WAL state before it can safely resume serving reads as a standby.
- Performance standby is purely a request-handling and read-routing feature, independent of the storage backend and of fault-tolerance mechanics - it scales reads, not writes, since the active node remains the single write bottleneck. For scaling writes or serving multiple regions at low latency, Performance Replication, which links separate clusters, is the complementary Enterprise feature.