AwsCLI icon
Docker icon
Log icon
If icon
Fail icon
Scan icon
SlackIncomingWebhook icon
Pause icon
Sleep icon
Schedule icon

Backfill the TTL attribute DynamoDB needs before it can expire anything

Detect DynamoDB items with a missing or String TTL attribute, backfill it as an epoch Number behind an approval, then re-read and fail if it did not land.

Categories
CloudDataInfrastructure

DynamoDB expires an item only when its TTL attribute exists and holds an epoch timestamp as a Number. An item that never got the attribute is skipped forever. So is one that got it as a String. TTL has no error channel for a skipped item, so nothing reports it. The table keeps growing. A retention promise the team believes is automated silently never runs.

This blueprint finds those items, works out the expiry each one should already have had, writes it, then reads the table again and fails the run if the repair did not land.

This blueprint was created by zkasuran.

Findings

Two plugin facts decided how the write is done, both measured on plugin-aws 2.8.7 against a real table.

  • io.kestra.plugin.aws.dynamodb.PutItem writes every value as a String. A JSON number in item and an unquoted integer in a YAML map both land as {"S": "1799999999"}. So PutItem cannot write the Number attribute TTL needs. Using it here would store the exact defect this flow exists to remove.
  • io.kestra.plugin.aws.dynamodb.Scan returns Number and List attributes as nothing at all. Its unmarshaller handles NULL, BOOL, String Set, Map and String, so a numeric created_at and a list attribute simply vanish from rows. Reading an item from Scan and writing it back whole would delete those attributes.

The flow therefore uses Scan for what it is exact at, counting the items that match a server side filter. The read and the write go through io.kestra.plugin.aws.cli.AwsCLI. The repair is a single attribute UpdateItem, not a full item overwrite, so no other attribute is touched and a concurrent update to the rest of the item is not lost. A full item overwrite would have been last writer wins.

Safety

  • Nothing is written unless auto_remediate is true, dry_run is false and a human resumes the Pause with approved: true. An unanswered Pause reads as a refusal.
  • Each write needs attribute_exists(<key>) and attribute_not_exists(<ttl>) or attribute_type(<ttl>, :s), so an item somebody else fixed or deleted between the scan and the write is skipped, never overwritten or recreated.
  • An item with no usable creation timestamp is reported by key, as is a String TTL that is neither an epoch nor a date. Neither is guessed. Writing an arbitrary TTL would schedule a delete nobody chose.
  • The detection filter has no expressionAttributeNames through this plugin, so ttl_attribute must not be a DynamoDB reserved word.
  • A repaired expiry that is already in the past is counted and reported separately. The flow does not push it forward, because that would quietly extend a retention window.

How it works

  1. read_ttl_config calls DescribeTimeToLive and DescribeTable through io.kestra.plugin.aws.cli.AwsCLI. It records whether TTL is enabled, which attribute it reads plus the table's key schema.
  2. refuse_misconfigured_ttl fails the run when TTL is disabled or configured on a different attribute. Writing an attribute the table does not read would expire nothing while looking like a fix.
  3. detect_unexpiring scans with io.kestra.plugin.aws.dynamodb.Scan and the filter attribute_not_exists(<ttl>) or attribute_type(<ttl>, :s), so it catches both a missing attribute and one stored as a String.
  4. plan_backfill re-reads the matching items as raw DynamoDB JSON and classifies each one. A missing attribute gets created_at + retention_days. A String that parses as an epoch or an ISO 8601 date keeps that same instant, converted to a Number, so no expiry date is invented. Anything else is refused by name. The plan is capped by max_items_per_run and sorted oldest expiry first.
  5. alert_findings posts the counts to Slack whenever the detector fires, in every mode.
  6. remediate_gate only opens with auto_remediate true and dry_run false. hold_for_approval then pauses for a human. An unanswered pause reads as a refusal.
  7. backfill_ttl writes one attribute per item with UpdateItem under the same condition, so an item somebody else already fixed or deleted is skipped.
  8. reverify_scan re-runs the detection scan and reverify_keys reads every touched item back with a consistent read. fail_if_unrepaired fails the run if any of them still has no Number TTL or if any write errored.

What you get

  • A count of the items your TTL setting can never act on, broken down into missing attribute and wrong type.
  • A repair that derives each expiry from data already in the item instead of inventing one.
  • A named list of items the flow refused to touch, so the undecidable cases reach a human.
  • A re-read that fails the run when the write did not land, so a green execution means the table really changed.
  • A read-only default plus an approval gate, so the first run cannot delete anything.

Who it's for

  • Platform and data teams running DynamoDB tables with TTL enabled.
  • Anyone holding a deletion deadline under GDPR, CCPA or an internal retention policy who needs proof the expiry is armed.
  • Teams whose storage bill grew after a writer shipped the TTL attribute as a String.

Why orchestrate this with Kestra

The repair needs a strict order: confirm the TTL setting, scan, classify, pause for approval, write, then read back. Kestra gives that order, a schedule, one audit log per execution, inputs that keep the default read-only plus a failure path that alerts. A script would run the same calls but would not leave an approval record or stop at the gate.

Tested end to end

Run against LocalStack DynamoDB on Kestra 2.0.5. The table held nine items with TTL enabled on expires_at: two with a valid Number TTL, three with no TTL attribute, three carrying it as a String, one with no creation timestamp at all. Ten executions covered every branch.

  • Clean table, three items all carrying a Number TTL: detector reported 0 and no alert fired.
  • Degraded table, dry_run: true: 7 of 9 detected, 4 missing the attribute, 3 String typed, 5 planned, 2 refused. aws dynamodb scan showed the table byte for byte unchanged afterwards.
  • TTL disabled on the table: refused and failed before any scan.
  • TTL enabled on expires_at while ttl_attribute was ttl_epoch: refused. Without that check the run would have written 7 items that still never expire.
  • Approval declined: nothing written, the five planned keys listed as still unexpirable.
  • Approval granted: 5 of 5 written, population 7 before and 2 after, the two survivors being exactly the refused pair. get-item confirmed each new value as {"N": ...}, confirmed the two healthy items untouched, then confirmed an item carrying a Map, a List and a Boolean kept all of them.
  • Re-run after the repair: 2 detected, 0 planned, no Pause, no write.
  • Concurrent writer overwrote a repaired item during the settle window: the re-read caught it, the run failed with the key named, then the Slack errors block posted.
  • Another writer set a correct Number TTL on a planned item during the Pause: that write was skipped on the condition expression and the other value survived.
  • Unreachable endpoint: the run failed on the first call and the Slack errors block posted.

Prerequisites

  • A DynamoDB table with TTL enabled, plus an item attribute holding a creation timestamp.
  • An IAM principal with dynamodb:DescribeTimeToLive, dynamodb:DescribeTable, dynamodb:Scan, dynamodb:GetItem and dynamodb:UpdateItem on the table.
  • Docker available to the worker, because the AWS CLI tasks run in a container.

Secrets

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • SLACK_WEBHOOK_URL

Inputs

  • dry_run (BOOL, default true) report only.
  • auto_remediate (BOOL, default false) allow the write.
  • table_name (STRING, default sessions) table to audit.
  • ttl_attribute (STRING, default expires_at) the attribute DynamoDB reads TTL from.
  • created_at_attribute (STRING, default created_at) where the expiry is derived from.
  • retention_days (INT, default 90) window added to the creation timestamp.
  • max_items_per_run (INT, default 25) cap on repairs per run.
  • scan_limit (INT, default 500) items read per detection pass.
  • aws_region (STRING, default us-east-1).
  • endpoint_url (STRING, default empty) point it at DynamoDB Local or LocalStack to rehearse.
  • access_key_id, secret_access_key (STRING, default empty) override the secrets for a rehearsal.
  • settle_duration (DURATION, default PT2S) wait before the re-read.

Quick start

  1. Add the three secrets.
  2. Run it with the defaults. It only reads. It prints how many items TTL cannot expire plus which ones it refuses to guess for.
  3. Check retention_days against your real policy, then look at the already past retention count in the report. Those items expire as soon as the attribute lands, which is the point, so make sure that is what you want.
  4. Re-run with auto_remediate: true and dry_run: false, read the plan in the paused execution, then resume with approved: true.
  5. Enable the weekly schedule once the counts look right.

How to extend

  • Point created_at_attribute at whatever your items really carry, for example inserted_at or an epoch ts.
  • Raise max_items_per_run once you trust the plan. Leaving it low lets the schedule drain the backlog a slice at a time.
  • Add a second detector for a TTL value more than five years in the past, which DynamoDB also refuses to act on.
  • Swap the Slack task for PagerDuty or Opsgenie if the refused list needs a ticket rather than a message.
  • Send the plan to S3 before the write if you need an auditable record of what was armed and when.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.