Webhook icon
Schedule icon
If icon
Build icon
Commands icon
Docker icon
Upload icon
List icon
Return icon
Fail icon
Download icon
Set icon
SlackIncomingWebhook icon

Prove Your etcd Snapshots Restore, Revision for Revision

Restore the latest etcd snapshot from S3 into a scratch member, check RPO, hash and quota, and compare it with the live cluster by HashKV.

Categories
Infrastructure

An etcd backup job that "succeeds" every night proves only that a file was uploaded. The file can be cut short by a failed multipart upload, taken from a member that had silently diverged, too old because the job died weeks ago, from the wrong cluster after a copy-paste in the bucket path, or too big to start under the backend quota. All of these surface on the day you need the restore, when the Kubernetes control plane is already down.

This blueprint restores the snapshot for real, every time: into a throwaway single-member etcd on random local ports inside the task container, never touching the live cluster. It then compares that member with the live cluster at the snapshot's revision using HashKV, etcd's own hash of every key revision, so a passing drill means the restored keyspace is identical to what the cluster held when the snapshot was taken, not "roughly the same number of keys". The demo runs with a local three-member cluster and SeaweedFS in Docker.

How it works

  1. build_tools (docker.Build) builds Alpine with the jq and etcd binaries copied from their official images, so the build needs no package mirror. The official etcd image is distroless, so script tasks cannot run in it.
  2. cluster_status (shell.Commands) asks every member for its status: leader, revision, database size, and active alarms (NOSPACE, CORRUPT). No leader stops the drill.
  3. fresh_snapshot (If, off by default) saves a snapshot from the leader and uploads it with its sha256 in the object metadata, for teams that want the flow to be the backup job as well.
  4. list_snapshots (aws.s3.List) and pick_snapshot choose the object named in the webhook body or input, else the newest one by lastModified, using the jq Pebble filter.
  5. download_snapshot (aws.s3.Download), then drill:
    • RPO: the object's age against rpo_hours.
    • The file itself, before anything parses it: etcdctl snapshot save appends a sha256 to the file, and bbolt files are whole 512-byte pages, so the size tells a hashed snapshot, a raw copy of member/snap/db (no hash) and a file cut short apart. The hash is then verified. etcd uses the same rule when it restores.
    • etcdutl snapshot status, then etcdutl snapshot restore into a scratch data directory.
    • Starts the restored member with the live cluster's --quota-backend-bytes and measures how long until it serves (your restore time).
    • HashKV at the snapshot's revision on every live member and on the restored one. The live members must agree with each other (a member that diverged is reported too), and the restored member must match. When the live cluster compacted after the snapshot, the scratch member is compacted to the same point first, since HashKV covers the revisions after the compaction point.
    • A key sample (sample_keys): keys the live cluster has not changed since the snapshot's revision must have the same value. When the live cluster has compacted past the snapshot, this is the comparison that remains. If most sampled keys do not exist in the live cluster, the snapshot comes from another cluster.
  6. verdict (If): on a pass, kv.Set remembers the last good drill and Slack gets the numbers. On a problem, Fail ends the execution FAILED with every problem found, and the errors block sends one alert that says which backup not to rely on and when the last good drill was.

What you get

  • A Slack message per drill. These are from the test runs:
    • "PASSED: snapshots/etcd-b-rev2226.db (0.0 h old, RPO 24 h) restored and serving in 1 s: 1775 keys, 0.2 MiB, revision 2226 (40 revisions behind the live cluster). HashKV at revision 2226 is identical on the restored member and the 3 live members that answered. Sample: 194 of 200 keys identical, 6 changed and 0 deleted since the snapshot."
    • "FAILED at drill_failed: the sha256 etcd appended does not match the content (258080 bytes): bytes changed after etcd wrote the file, in storage or in transit"
    • "FAILED at drill_failed: the file is 129040 bytes, not whole 512-byte pages plus etcd's 32-byte hash: it was cut short (an interrupted upload or copy, a full disk on the backup host)"
    • "FAILED at drill_failed: 200 of 200 sampled keys do not exist in the live cluster: the snapshot is probably from another cluster, or most of the keyspace was deleted since"
    • "FAILED at drill_failed: the restored database is 16.0 MiB, at or above the 8 MiB backend quota: a member restored from it starts in the NOSPACE alarm."
  • outputs.result: ok, key, age_hours, snapshot_revision, revisions_behind, restored_keys, restored_db_mb, restore_seconds, compare (hashkv or sample), both hashes, the sample counts, problems and warnings.
  • The KV entry etcd_drill_last_good with the time, key and revision of the last snapshot that passed.

Error handling

  • Every finding is collected, so one run reports a stale and corrupt snapshot together, then the run ends FAILED.
  • Infrastructure failures (no member answers, no leader, S3 unreachable, the scratch member cannot start) fail the task that hit them with the reason, and the errors block names that task.
  • S3 calls retry with exponential backoff. The scratch member retries on a new random port if the first one is taken.
  • The drill only reads from the live cluster (endpoint status, alarm list, endpoint hashkv, get). snapshot save runs only with take_snapshot, against the leader.
  • concurrency: limit 1 queues a webhook call that arrives while the nightly run is still restoring.

What this blueprint teaches about Kestra

  • S3-compatible storage with the AWS plugin: endpointOverride and forcePathStyle point s3.List, s3.Download and s3.Upload at SeaweedFS, MinIO, Ceph or R2 as well as AWS.
  • docker.Build inside the flow, to add a shell to a distroless tool image.
  • A script that reports instead of failing: the drill emits a structured result whether it passes or not, and an If turns ok: false into a Fail. The outputs stay machine-readable for CI even when the drill fails, and the errors block sends the single alert.
  • The jq Pebble filter to pick an object from a task's output list, and trigger.body with ?? so the webhook and manual runs share one flow.
  • Optional secrets: {{ inputs.tls ? secret('ETCD_CA_PEM') : '' }} only reads the secret when TLS is on.

Prerequisites

  • A Kestra worker that can run Docker containers and reach the etcd client URLs. The shell tasks use networkMode: host, so they reach whatever the worker host reaches, including the demo cluster on 127.0.0.1.
  • Snapshots in an S3-compatible bucket, written by etcdctl snapshot save (RKE2, k3s, kubeadm backup jobs and most operators do this), or set take_snapshot.
  • A Slack incoming webhook.

Secrets

  • AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY: read access to the bucket (write too with take_snapshot).
  • SLACK_WEBHOOK_URL.
  • ETCD_DRILL_WEBHOOK_KEY: a long random string for the webhook.
  • With tls: true: ETCD_CA_PEM, ETCD_CERT_PEM, ETCD_KEY_PEM (on kubeadm clusters: /etc/kubernetes/pki/etcd/ca.crt and the apiserver-etcd-client pair).

Inputs

  • endpoints (every member's client URL), bucket, prefix, snapshot_key, s3_endpoint, region.
  • take_snapshot (false), rpo_hours (24), sample_keys (200), quota_backend_mb (2048), skip_hash_check (false), tls (false).
  • etcd_version (v3.5.17) and build_image (true).

Outputs

  • result (JSON): the drill result described above, also when the drill fails.

Quick start

  1. Start a three-member etcd and a local S3 (SeaweedFS) with one access key:
    for m in 1:2379 2:22379 3:32379; do n=${m%%:*}; c=${m#*:}; p=$((c + 1))
      docker run -d --name etcd$n --network host quay.io/coreos/etcd:v3.5.17 etcd --name e$n --data-dir /data \
        --listen-client-urls http://127.0.0.1:$c --advertise-client-urls http://127.0.0.1:$c \
        --listen-peer-urls http://127.0.0.1:$p --initial-advertise-peer-urls http://127.0.0.1:$p \
        --initial-cluster e1=http://127.0.0.1:2380,e2=http://127.0.0.1:22380,e3=http://127.0.0.1:32380
    done
    echo '{"identities":[{"name":"drill","credentials":[{"accessKey":"drill","secretKey":"drill-secret"}],"actions":["Admin","Read","Write","List"]}]}' > s3.json
    docker run -d --name s3 --network host -v $PWD/s3.json:/s3.json chrislusf/seaweedfs server -s3 -s3.port=8333 -s3.config=/s3.json -volume.port=8088 -volume.max=5 -master.volumeSizeLimitMB=64 -dir=/data -ip=127.0.0.1
    sleep 10; docker run --rm --network host -e AWS_ACCESS_KEY_ID=drill -e AWS_SECRET_ACCESS_KEY=drill-secret -e AWS_DEFAULT_REGION=us-east-1 \
      amazon/aws-cli --endpoint-url http://127.0.0.1:8333 s3 mb s3://etcd-backups
    
  2. Add the secrets (AWS_ACCESS_KEY_ID=drill, AWS_SECRET_ACCESS_KEY=drill-secret for this demo), put a few keys in etcd (etcdctl put /registry/pods/default/a 1), then run once with take_snapshot checked. The next runs drill the newest snapshot in the bucket.
  3. Real result from the test cluster (1,775 keys): PASSED, restored and serving in 1 s, HashKV identical on the restored member and the 3 live members.
  4. Point endpoints, bucket, prefix and s3_endpoint at your cluster and backups, set tls, and call the webhook from your backup job.

How to read the result

  • revisions_behind counts the writes the cluster accepted after the snapshot. They are what a restore loses; on Kubernetes, controllers recreate most of them, but not everything (leases, events, recently created objects).
  • compare: sample is normal when the live cluster compacts often (kube-apiserver compacts every 5 minutes). Drill the snapshot right after the backup job writes it, through the webhook, to keep the exact HashKV comparison.
  • "live members disagree" is about the cluster, not the backup: one member's data has diverged. Find it with etcdctl endpoint hashkv --cluster, remove and re-add it.

How to extend

  • Restore with the same member layout as production (three scratch members) to also measure the time to a new quorum.
  • Write the drill result to a table with the JDBC plugin to chart restore time and snapshot size over months.
  • Chain the drill to your snapshot retention job, so old snapshots are only deleted after a newer one has passed.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.