Automatically Ship Workflow and System Logs to AWS CloudWatch

Centralize your Kestra workflow and system logs in AWS CloudWatch for production-grade observability, auditing, and incident investigation. This blueprint uses the Kestra Enterprise Edition log shipper to collect orchestration logs over a configurable lookback window, filter them by severity, and push them into a dedicated CloudWatch log group and stream. Instead of scraping logs from scattered execution views, you get a single, queryable destination wired into your existing AWS monitoring stack, complete with state tracking so events are exported once and never duplicated.

How it works

• A io.kestra.plugin.core.trigger.Schedule trigger named daily runs the flow once a day on the @daily cron.
• The log_export task (io.kestra.plugin.ee.core.log.LogShipper) collects logs at or above INFO level over a P1D lookback period.
• An offsetKey (log_shipper_aws_cloudwatch_state) persists the export position so each run picks up only new logs, and delete: false keeps the source logs in place after shipping.
• The configured io.kestra.plugin.ee.aws.cloudwatch.LogExporter writes those logs to the CloudWatch log group kestra and log stream kestra-log-stream in the region set by vars.region.

What you get

• Workflow and system logs centralized in AWS CloudWatch Logs.
• Severity filtering so only logs at the level you choose are exported.
• Exactly-once export via offset tracking, avoiding duplicates across runs.
• A foundation for CloudWatch alarms, dashboards, and downstream observability tooling.

Who it's for

• Platform and SRE teams running Kestra in production on AWS.
• DevOps engineers consolidating logs into a single observability backend.
• Compliance and audit owners who need durable, retained log records.

Why orchestrate this with Kestra

Kestra turns log shipping into a declarative, scheduled, and observable pipeline defined entirely in YAML. The Schedule trigger drives recurring exports, retries and execution history make failures visible and recoverable, and offset state guarantees no gaps or duplicates between runs. CloudWatch can store and alarm on logs, but it cannot reach into Kestra to pull orchestration and system logs on a cadence with filtering and offset bookkeeping. This blueprint fills that gap and keeps the export logic versioned alongside the rest of your workflows.

Prerequisites

• A Kestra Enterprise Edition instance (this flow uses EE-only log shipper tasks).
• An AWS account with a CloudWatch Logs log group and stream, plus IAM permissions to write log events.
• A region variable set to your target AWS region.

Secrets

• AWS_ACCESS_KEY_ID: access key for an IAM identity allowed to write to CloudWatch Logs.
• AWS_SECRET_KEY_ID: corresponding AWS secret access key.

Quick start

1. Add this flow to a Kestra Enterprise Edition instance.
2. Store AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID as secrets.
3. Set the region variable and confirm the kestra log group and kestra-log-stream stream exist in CloudWatch.
4. Enable the daily schedule, or trigger a manual run to validate the export end to end.

How to extend

• Adjust logLevelFilter (for example to WARN) to control which severities are shipped.
• Change lookbackPeriod or the schedule cron to match your retention and freshness needs.
• Add more exporters under logExporters to fan logs out to additional destinations alongside CloudWatch.
• Set delete: true to purge source logs after a successful export when you want CloudWatch to be the single source of truth.

Links

• AWS plugin documentation
• Orchestrate AWS with Kestra