Autoscale EMR Serverless Spark on a CloudWatch Backlog

Run Spark on AWS EMR Serverless only when there is real work to process. This blueprint turns a CloudWatch backlog metric, such as an SQS queue depth, into a scaling signal: on a fixed schedule it samples the metric, and when datapoints show pending work it submits a Spark job to an existing EMR Serverless application, then reports the run to Slack. It solves the classic gap between a time based scheduler and demand based compute, so you stop paying for idle Spark capacity and stop running empty jobs on a clock that ignores the actual backlog.

How it works

1. The every_15_min Schedule trigger (io.kestra.plugin.core.trigger.Schedule, cron */15 * * * *, shipped disabled) starts the flow on a cadence.
2. read_backlog (io.kestra.plugin.aws.cloudwatch.Query) reads the AWS/SQS namespace metric named by backlog_metric, using the Maximum statistic over a PT15M window at 300 second periods.
3. scale_if_backlog (io.kestra.plugin.core.flow.If) evaluates {{ outputs.read_backlog.count > 0 }} and only proceeds when the metric returned datapoints.
4. start_spark_job (io.kestra.plugin.aws.emr.StartServerlessJobRun) submits the Spark job to the application_id you provide, assuming execution_role_arn, naming the run kestra-backlog-{{ execution.id }}, with entry_point pointing at your S3 script.
5. notify (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) posts the returned jobRunId to Slack.

What you get

• Demand driven Spark runs that fire only when the backlog is above zero.
• No idle EMR Serverless compute and no empty jobs triggered purely by the clock.
• A Slack message per submitted run for visibility.
• A single declarative YAML flow you can version and review.

Who it's for

• Data engineers running batch Spark on EMR Serverless.
• Platform teams who want event or demand based scaling instead of fixed cron jobs.
• Anyone bridging SQS or CloudWatch signals to AWS compute.

Why orchestrate this with Kestra

EMR Serverless scales executors within a job, but it has no native scheduler that inspects an external backlog before deciding whether to launch a job at all. Kestra fills that gap: the Schedule trigger drives the cadence, the If task gates submission on a live CloudWatch reading, retries and replays cover transient AWS errors, and every run is captured with full execution lineage and outputs. The whole pipeline is declarative YAML, so the scaling logic lives in source control rather than scattered across cron entries and Lambda glue.

Prerequisites

• An existing EMR Serverless application and an IAM execution role ARN.
• A Spark entry point uploaded to S3.
• A CloudWatch metric that represents your backlog (default ApproximateNumberOfMessagesVisible in AWS/SQS).

Secrets

• AWS_ACCESS_KEY_ID: AWS access key.
• AWS_SECRET_ACCESS_KEY: AWS secret key.
• AWS_DEFAULT_REGION: AWS region.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the secrets above to your Kestra namespace.
2. Provide application_id, execution_role_arn, and entry_point as inputs.
3. Point read_backlog at the namespace and metric that represent your queue depth.
4. Enable the every_15_min trigger when you are ready to run on a schedule.

How to extend

• Swap the AWS/SQS namespace and metric for any CloudWatch signal (Kinesis, custom metrics, application gauges).
• Raise the If condition past zero to require a minimum backlog before scaling.
• Add sparkSubmitParameters or job arguments to start_spark_job for tuned Spark configs.
• Chain a polling task or EMR event listener after submission to wait for completion before notifying.

Links

• Plugin docs: https://kestra.io/plugins/plugin-aws
• Orchestration: https://kestra.io/orchestration/aws