Amazon DynamoDB Cold Storage Archival and Purge to S3

Amazon DynamoDB is built for low-latency operational workloads, but storing historical, soft-deleted, or expired records in primary tables indefinitely leads to unbounded storage expenses. While DynamoDB table storage costs $0.25 per GB-month, Amazon S3 Standard costs $0.023 per GB-month (over 10x cheaper) and archival tiers like S3 Glacier Deep Archive cost less than $0.001 per GB-month (over 250x cheaper). Furthermore, DynamoDB's native Time-to-Live (TTL) feature guarantees deletion only within 48 hours of expiration, which can cause downstream queries to return stale data or violate strict compliance windows.

This blueprint implements an automated, production-grade retention tiering and garbage collection pipeline for DynamoDB. It scans records matching an ISO-8601 retention cutoff, writes them into internal storage manifests, safely archives them into an Amazon S3 cold bucket under date-partitioned keys, and removes the purged records from the primary DynamoDB table only after S3 write durability is confirmed. The workflow features safe-by-default dry-run controls, serialized execution concurrency to prevent race conditions, and immediate Slack alerting on task failures.

Table Schema & Setup

This blueprint assumes items in your DynamoDB table contain an id partition key and an ISO-8601 UTC timestamp string attribute representing record expiration or soft deletion:

{
  "id": "evt_01HZX87Q9E1K4B9A2M4C5D6E7F",
  "event_type": "ORDER_COMPLETED",
  "customer_id": "cust_90214",
  "expires_at": "2026-07-14T03:00:00Z",
  "payload": {
    "order_id": "ord_88201",
    "total_amount": 128.50
  }
}

How it works

1. Execution Serialization: Flow-level concurrency: { behavior: QUEUE, limit: 1 } prevents concurrent execution overlap between scheduled cron sweeps and manual on-demand webhook invocations.
2. Guarded Retention Input: The retentionDays input enforces min: 0 to prevent negative offsets from advancing the cutoff into future timestamps, while filterAttribute validates against ^[a-zA-Z_][a-zA-Z0-9_]*$ to reject malformed expressions or invalid attribute names.
3. Targeted Expiration Scan: scan_expired_records (io.kestra.plugin.aws.dynamodb.Scan) evaluates the table with a filter expression comparing the configured expiration attribute against :cutoff. The cutoff timestamp is computed dynamically in Kestra using {{ now() | dateAdd(-inputs.retentionDays, 'DAYS') | date("yyyy-MM-dd'T'HH:mm:ss'Z'", timeZone="UTC") }}.
4. Short-Circuit Backlog Evaluation: evaluate_archival_batch (io.kestra.plugin.core.flow.If) evaluates outputs.scan_expired_records.size > 0. If no records exceed the retention threshold, the workflow logs a clean status and completes without executing redundant S3 uploads or downstream loops.
5. Archive Manifest Generation: write_archive_manifest (io.kestra.plugin.core.storage.Write) converts the matched records into an internal JSON storage file.
6. Durable Cold Storage Upload: upload_to_s3_archive (io.kestra.plugin.aws.s3.Upload) pushes the manifest into the target Amazon S3 bucket under the key {{ inputs.s3Prefix }}/{{ inputs.tableName }}/{{ execution.startDate | date('yyyy/MM/dd') }}/{{ execution.id }}.json.
7. Dry-Run Gate & Item Pruning: check_dry_run_and_purge (io.kestra.plugin.core.flow.If) verifies dryRun: false. It executes purge_items_from_dynamodb (io.kestra.plugin.core.flow.Loop) over the archived records and sequentially invokes delete_item (io.kestra.plugin.aws.dynamodb.DeleteItem) by primary key (id). Because deletions occur strictly downstream of upload_to_s3_archive, records are never deleted unless S3 persistence has succeeded.
8. Audit Metrics: log_purge_summary and log_dry_run_summary (io.kestra.plugin.core.log.Log) output structured confirmation logs with item counts and destination S3 URIs.
9. Operational Escalation: notify_archival_incident in the flow-level errors block dispatches a Slack alert with execution and task failure context if any step fails.

What you get

• Up to 90%+ storage cost savings by tiering cold DynamoDB data to Amazon S3 Standard or Glacier.
• Zero data loss guarantee: records are deleted from DynamoDB only after S3 write durability is confirmed.
• Safe-by-default operations with dryRun: true, disabled: true on the schedule trigger, and min: 0 retention bounds.
• Automated execution serialization with QUEUE concurrency to prevent race conditions.
• Immediate incident alerting through Slack Incoming Webhooks.
• Complete execution audit trails and artifact history in Kestra.

Who it's for

• Cloud and Platform Engineers managing AWS infrastructure and cloud spend.
• Data Engineers maintaining NoSQL operational stores, compliance retention, and data lakes.
• Site Reliability Engineers establishing automated, resilient database maintenance runbooks.

Why orchestrate this with Kestra

Building custom retention tiering scripts with AWS Lambda or EC2 cron jobs introduces operational burdens: Lambda 15-minute execution timeouts, fragile IAM permission configurations, lack of native queueing/serialization, unhandled batch retries, and absent execution lineage. Kestra delivers:

• A native scheduler (Schedule) and authenticated Webhook trigger in a single YAML specification.
• Built-in QUEUE concurrency to eliminate overlapping workers without Redis locks or distributed mutexes.
• Transparent artifact storage lineage from DynamoDB scan to internal storage and final S3 destination.
• Declarative task-level retries and flow-level error handlers paging Slack out of the box.

Prerequisites

• An active AWS account with permissions for Amazon DynamoDB (dynamodb:Scan, dynamodb:DeleteItem) and Amazon S3 (s3:PutObject).
• An Amazon DynamoDB table with an id primary key and an ISO-8601 timestamp attribute.
• An Amazon S3 bucket configured for cold storage retention.
• A Slack Incoming Webhook URL for error alerts.

Secrets

Configure the following secrets in your Kestra namespace:

• AWS_ACCESS_KEY_ID: IAM access key with DynamoDB and S3 permissions.
• AWS_SECRET_KEY_ID: Corresponding IAM secret key.
• AWS_DEFAULT_REGION: Target AWS region (e.g. us-east-1, eu-west-1).
• DYNAMODB_PURGE_WEBHOOK_KEY: Secret authentication key for the manual webhook trigger.
• SLACK_WEBHOOK_URL: Slack Incoming Webhook URL for incident notifications.

Quick start

1. Configure the required secrets (AWS_ACCESS_KEY_ID, AWS_SECRET_KEY_ID, AWS_DEFAULT_REGION, DYNAMODB_PURGE_WEBHOOK_KEY, SLACK_WEBHOOK_URL) in your Kestra namespace.
2. Adjust the default tableName, s3Bucket, and retentionDays inputs to match your AWS environment.
3. Trigger a manual execution with dryRun: true in the Kestra UI to verify that expired records are detected and uploaded to S3 without removing them from DynamoDB.
4. Verify the uploaded JSON object in your S3 bucket.
5. Set dryRun: false and toggle disabled: false on daily_retention_schedule to automate ongoing archival sweeps.

How to extend

• Parquet Conversion: Chain an io.kestra.plugin.jdbc.duckdb.Query task between the scan and upload to compress JSON manifests into columnar Parquet format before writing to S3.
• S3 Glacier Lifecycle: Configure AWS S3 Lifecycle rules on the target prefix to automatically transition archived objects from S3 Standard to Glacier Deep Archive after 30 days.
• Multi-Table Archival: Wrap the scan-and-archive tasks in a parent io.kestra.plugin.core.flow.Parallel task or subflow to sweep multiple application tables concurrently.

Links

• Kestra AWS Plugin Documentation
• Amazon DynamoDB Documentation
• Amazon S3 Documentation
• This blueprint was created by Vedant817