Extract and process data from DynamoDB

Pull data out of an Amazon DynamoDB table and feed it straight into a processing step. This blueprint runs a full table scan against DynamoDB, returns the matching items as structured rows, and passes them to a shell task so you can transform, validate, export, or load the records anywhere your pipeline needs them. It closes the common gap between a NoSQL key-value store and the rest of your data stack, where DynamoDB has the data but no built-in way to schedule extraction and route it downstream.

How it works

1. The extract_data task (io.kestra.plugin.aws.dynamodb.Scan) scans the table named demo with fetchType: FETCH, so every matching item is returned inline as outputs.extract_data.rows. Region and credentials are injected from secrets via region, accessKeyId, and secretKeyId.
2. The process_data task (io.kestra.plugin.scripts.shell.Commands) runs on the io.kestra.plugin.core.runner.Process task runner and consumes {{ outputs.extract_data.rows }}, the JSON payload produced by the scan.

What you get

• A repeatable DynamoDB extraction you can run on demand or on a schedule.
• Scanned items exposed as a reusable output that downstream tasks can consume directly.
• Credentials kept out of the flow definition through Kestra secrets.
• A clear two-step pattern (extract, then process) you can extend into a full ELT pipeline.

Who it's for

• Data engineers moving DynamoDB tables into warehouses, lakes, or analytics tools.
• Platform teams who need scheduled, auditable exports from operational NoSQL stores.
• Developers building event-driven syncs between DynamoDB and other systems.

Why orchestrate this with Kestra

DynamoDB stores data but has no native scheduler, retry policy, or way to chain extraction into downstream processing. Kestra adds event and schedule triggers, automatic retries on transient AWS errors, full execution lineage and logs for every scan, and a declarative YAML definition you can version control and review. The result is a reliable, observable extraction pipeline instead of an ad hoc script that no one can monitor.

Prerequisites

• An existing DynamoDB table (the flow scans demo, change tableName to your own).
• An AWS IAM principal with permission to scan the target table.

Secrets

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_DEFAULT_REGION

Quick start

1. Add the three AWS secrets above to your Kestra namespace or environment.
2. Set tableName on extract_data to the table you want to scan.
3. Run the flow and inspect the extract_data outputs to confirm the rows come back.
4. Adjust the process_data commands to do something useful with the data.

How to extend

• Swap the shell process_data step for a load task that writes to S3, Postgres, BigQuery, or Snowflake.
• Add a schedule or event trigger to run the scan automatically.
• Filter or limit the scan, or page through large tables, to control payload size.
• Add retries and alerting on failure for production-grade reliability.

Links

• Plugin docs: https://kestra.io/plugins/plugin-aws
• Orchestrate AWS with Kestra: https://kestra.io/orchestration/aws