Add a job step to an existing AWS EMR cluster

Source

yaml

id: add-aws-emr-job
namespace: company.team

tasks:
  - id: ad_step_cluster
    type: io.kestra.plugin.aws.emr.SubmitSteps
    accessKeyId: "{{ secret('AWS_ACCESS_KEY') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY') }}"
    region: eu-west-3
    clusterId: "j-3GMIV2CAUOFR6"
    steps:
      - name: Spark_job_test
        jar: "command-runner.jar"
        actionOnFailure: CONTINUE
        commands:
          - spark-submit s3://kestra-test/health_violations.py --data_source
            s3://kestra-test/food_establishment_data.csv --output_uri
            s3://kestra-test/test-emr-output

About this blueprint

AWS

This flow adds a job step to an AWS EMR cluster and runs a Spark Job.

Submit Steps

More Related Blueprints

Ingest AWS

Add a new item to a DynamoDB table

AWS Python Task Runner

Run a Python script on AWS ECS Fargate with AWS Batch

New to Kestra?

Use blueprints to kickstart your first workflows.

Get started with Kestra