HashiCorp Consul Associate (003) Study Guide
The HashiCorp Consul Associate (003) exam validates that you can use Consul for service networking - service discovery, health checking, the key/value store, service mesh (Connect), and core security. It is a 60-minute, multiple-choice exam aimed at developers, operators, and architects who deploy and operate Consul in production and want to prove foundational competence.
Domain 1: Consul Architecture
- Consul is a single binary that runs as an agent in one of two modes: server (participates in Raft consensus and stores cluster state) or client (lightweight, registers local services, runs health checks, and forwards RPCs/queries to servers).
- Server quorum should be an odd number of voting servers - typically 3 (tolerates 1 failure) or 5 (tolerates 2 failures) - so Raft can elect a leader and commit writes without split-brain.
- Losing quorum (e.g. 2 of 3 servers down) means Consul cannot elect a leader or commit writes; reads of stale data may still work but no new writes are accepted.
- Raft uses a single leader; all writes go to the leader, are replicated to a majority quorum, and must be durably fsync'd to disk before being acknowledged - so disk I/O latency directly limits write throughput.
- Adding more voting servers increases write latency because every commit must be replicated to and acknowledged by a larger quorum.
- Consul uses the Serf gossip protocol for membership and failure detection: a LAN gossip pool within a datacenter and a WAN gossip pool between federated datacenters.
- A datacenter is a grouping of Consul agents/servers forming one cluster (often one per region/site); multiple datacenters can be federated for cross-DC discovery while each keeps its own local Raft quorum.
- performance.raft_multiplier scales internal Raft timing; the default is 1 for production, and a higher value (e.g. 5) relaxes timeouts for slower/dev hardware at the cost of slower leader failover.
- Consul Enterprise read scaling uses non-voting servers (-non-voting-server) that replicate the log and serve stale reads but do not participate in quorum, so they add read capacity without raising write latency.
- Consul computes network coordinates (network tomography) to estimate round-trip time; consul rtt reports RTT between nodes and prepared queries can sort results by nearest node.
- consul operator raft list-peers lists the current Raft peers and their voter/leader status; consul validate validates configuration files before starting an agent.
- Autopilot (servers) automatically cleans up failed/dead servers, manages stable server introduction, and on Enterprise supports redundancy zones and automated upgrades.
- Take regular state snapshots for disaster recovery with consul snapshot save; snapshots capture KV, sessions, ACLs, and the service catalog for point-in-time restore.
- The control plane is the set of server agents running Raft; the data plane is the sidecar (Envoy) proxies that actually carry application traffic in the service mesh.
Domain 2: Service Discovery
- Services are registered with Consul via a service definition file (HCL/JSON in the agent config directory), the HTTP API (/v1/agent/service/register), or consul services register web.hcl.
- By default Consul's DNS interface listens on port 8600; query <service>.service.consul to get addresses of healthy instances and <service>.service.consul SRV for host+port.
- Only passing (healthy) instances are returned in discovery results by default; an instance that fails its health check is excluded until it passes again.
- Consul supports multiple check types: HTTP, TCP, gRPC, script, Docker, and TTL checks where the application must periodically report passing or be marked critical.
- deregister_critical_service_after automatically removes a service that has been in the critical state for a configured duration, cleaning up dead instances from the catalog.
- The HTTP health endpoint /v1/health/service/<name>?passing returns only healthy instances; without ?passing it returns all instances with their check status.
- Stale reads (allow_stale, default enabled for DNS) let any server answer queries from its local state without contacting the leader, trading slight staleness for higher read throughput.
- dns_config.service_ttl sets a positive DNS TTL for service lookups so resolvers can cache results and reduce query load on Consul.
- Prepared queries add features DNS alone cannot: a Failover stanza can redirect to other datacenters or nearest locality when no healthy local instances exist.
- Blocking queries use the X-Consul-Index header and a wait parameter for efficient long-polling: Consul responds only when the result changes, avoiding fixed-interval polling.
- Client-side discovery (clients resolving healthy instances directly through Consul) removes the need for dedicated per-service load-balancer appliances.
- A health check definition looks like check { http = "http://localhost:8080/health", interval = "10s" }; tune interval and timeout to detect failures promptly without excessive load.
- Deregister a specific instance with consul services deregister -id=web1 or via the API; registration is node-local, so a service is removed if its node leaves the cluster.
- Service discovery decouples consumers from hard-coded IPs/ports, and routing to only-healthy instances improves overall reliability.
Domain 3: Consul KV and Configuration
- The Consul KV store holds dynamic configuration, feature flags, and coordination data, letting apps update behavior centrally without redeploying.
- Write a key with consul kv put app/maintenance true and read it with consul kv get app/maintenance; the API path is /v1/kv/<key>.
- consul kv get -recurse app/ returns all keys and values under a prefix in one call; consul kv delete -recurse app/ removes the whole subtree.
- Use a hierarchical key namespace (e.g. app/env/setting) so ACL tokens can be scoped to specific prefixes for least-privilege access.
- Check-And-Set (CAS) writes use the key's ModifyIndex: consul kv put -cas -modify-index=<n> key value succeeds only if the key has not changed, giving optimistic concurrency.
- Consul sessions plus KV CAS enable distributed locks and leader election (a session acquires a key, and the lock releases if the session or its TTL expires).
- Blocking queries on KV use ?index=<X> with an optional &wait=<duration> so Consul replies only when the value changes - the basis for watches and consul-template.
- Watches and consul-template let you trigger actions (regenerate config, restart a service) when KV values or service health change.
- KV transactions (the /v1/txn API) apply multiple KV operations atomically so several keys are updated together or not at all.
- Do not store secrets in plaintext in KV - use HashiCorp Vault and reference secrets from it; KV is not encrypted at rest by Consul itself.
- Every KV write goes through Raft and is replicated and snapshotted, so large values bloat the Raft log, snapshots, and replication cost; values have a default 512 KB size limit.
- Periodically prune unused keys/prefixes and avoid storing large or transient data in KV to keep the Raft log and snapshots small.
- KV access is controlled by ACL policies granting read/write on specific key paths; tokens scoped to prefixes enforce who can change which configuration.
- Prefer blocking queries/watches over fixed-interval polling to react to configuration changes efficiently and reduce load on the servers.
Domain 4: Service Mesh and Security
- Consul service mesh (Connect) provides automatic mutual TLS (mTLS) and service-to-service authorization without requiring changes to application code.
- Each service runs a sidecar proxy (Envoy is the supported proxy) that transparently establishes mTLS and enforces authorization, keeping the application unaware of mesh security.
- Intentions define which services are allowed (or denied) to communicate; create one with consul intention create web db to permit web to call db.
- Set a default-deny posture so services cannot talk to each other unless an intention explicitly allows it (zero-trust networking), independent of network topology.
- Connect issues a short-lived leaf (identity) certificate to each service from a CA; the built-in CA is the default, and Vault can be configured as the Connect CA provider for production.
- Longer leaf-certificate lifetimes are signed and rotated less frequently, reducing CA signing load and proxy churn (at the cost of slower credential rotation).
- Start a sidecar with consul connect envoy -sidecar-for web; the proxy registers as a connect-proxy for the named service.
- Transparent proxy mode redirects traffic to the sidecar automatically and, combined with intentions/explicit upstreams, makes Consul send proxies only the relevant upstream endpoints.
- proxy-defaults and service-defaults config entries (written with consul config write file.hcl) set mesh-wide and per-service defaults like protocol and connection/pending-request limits.
- Locality-aware routing/failover keeps traffic in the same zone when possible, improving latency and reducing cross-AZ data-transfer charges.
- L7 traffic management config entries - service-router, service-splitter, and service-resolver - enable path/header routing, weighted traffic splitting (canary), and failover.
- Intentions support identity-based authorization at L4 (allow/deny) and, when the protocol is set to http, finer-grained L7 rules on paths, methods, and headers.
- Service mesh adds capabilities like circuit breaking and observability; tuning trace sampling and metric cardinality controls observability overhead and backend storage spend.
- Mesh gateways enable secure mTLS connectivity between services across datacenters or networks without exposing services directly or requiring full IP connectivity.
Domain 5: Security and Architecture
- Consul ACLs provide token-based access control over resources (services, KV, nodes, sessions); enable them with acl { enabled = true, default_policy = "deny" } for least privilege.
- With default_policy = deny, all access is denied unless a token grants it via attached policies (allowlist model); default_policy = allow is permissive and not recommended for production.
- The bootstrap (initial management) token is created once with consul acl bootstrap; it is a global management token with unrestricted privileges, so guard and rotate it carefully.
- An ACL token has a SecretID (the bearer credential sent in X-Consul-Token requests) and an AccessorID (a non-secret identifier used for logging, auditing, and token management).
- Create least-privilege tokens by attaching policies, e.g. consul acl token create -policy-name=web-read; assign separate agent, service, and session tokens scoped to what each needs.
- Auth methods (e.g. Kubernetes, JWT/OIDC) let workloads obtain dynamic, short-lived tokens automatically instead of hand-managing static tokens.
- Gossip encryption uses a shared symmetric key for the Serf LAN/WAN pools; generate it with consul keygen and install/rotate keys with consul keyring -install <key>.
- TLS (mTLS) on the RPC/control plane authenticates agents via certificates, preventing untrusted nodes from joining or impersonating cluster members and encrypting agent-to-server traffic.
- Federated/secondary datacenters replicate ACL tokens and policies and cache them locally, so each DC keeps a local Raft quorum while sharing identity and discovery across DCs.
- Take regular backups with consul snapshot save backup.snap (or the Enterprise snapshot agent) and test restores with consul snapshot restore; snapshots include KV, ACLs, sessions, and catalog.
- raft_snapshot_threshold and raft_snapshot_interval control how often Raft compacts its log into a snapshot, trimming log growth on busy clusters.
- Use autopilot to automatically remove failed/dead servers so the Raft peer set does not become bloated with stale members.
- Use HashiCorp Vault for dynamic, short-lived secrets and optionally as the Connect CA, keeping sensitive material out of Consul KV.
- Defense in depth combines gossip encryption, RPC mTLS, ACLs with default-deny, and mesh intentions - each layer protects a different surface of the cluster.
HashiCorp Consul Associate (003) exam tips
- Memorize the quorum/fault-tolerance table: 3 servers tolerate 1 failure, 5 tolerate 2; always use an odd number of voting servers and know that more servers means higher write latency.
- Be fluent in the core CLI verbs and their flags - consul kv get/put -recurse and -cas, consul services register/deregister, consul intention create, consul acl token create, consul snapshot save, and consul operator raft list-peers.
- Distinguish the three security layers and what each protects: gossip encryption (Serf membership traffic), TLS/mTLS (RPC control plane and service mesh data plane), and ACLs (authorization over resources).
- Know default behaviors and ports: DNS on 8600, only healthy instances returned by default, ACL default_policy deny vs allow, stale reads enabled for DNS, and blocking queries driven by X-Consul-Index.
- Read scenario questions carefully for the failure mode being described (lost quorum, critical health check, default-deny intention blocking traffic, expired session releasing a lock) and pick the answer that matches Consul's documented default.
Study guide FAQ
What is the difference between a Consul server agent and a client agent?
Server agents participate in Raft consensus, store and replicate all cluster state (catalog, KV, ACLs), and elect a leader, so you run a small odd number of them (3 or 5). Client agents are lightweight: they register local services, run health checks, take part in gossip, and forward queries/RPCs to the servers, but they hold no persistent state and do not affect quorum. This separation lets you have many clients without slowing down consensus.
How do service discovery, the KV store, and service mesh relate to each other?
They are layered features of the same platform. Service discovery registers services and returns only healthy instances via DNS or HTTP. The KV store holds dynamic configuration and coordination data. The service mesh (Connect) builds on discovery by adding sidecar proxies that provide automatic mTLS and intention-based authorization between services. You can use discovery and KV without the mesh, but the mesh relies on the underlying catalog and identity that discovery and ACLs provide.
What are intentions and why does default-deny matter?
Intentions are mesh authorization rules that declare which source services may connect to which destination services, enforced by the sidecar proxies regardless of network topology. Setting a default-deny posture means no service can communicate until an intention explicitly allows it, which implements zero-trust networking. This is the recommended production stance because it fails closed: a misconfiguration blocks traffic rather than silently allowing unauthorized access.
How do I keep my Consul cluster secure and recoverable?
Enable ACLs with default_policy = deny and issue least-privilege tokens (separate agent, service, and session tokens), turn on gossip encryption with a shared key, and enable TLS/mTLS so only certificate-authenticated agents can join. Store secrets in Vault rather than KV, and optionally use Vault as the Connect CA. For recoverability, schedule regular consul snapshot save backups (or the snapshot agent) and periodically test restores, and rely on autopilot to clean up dead servers.