AWS CreateServerlessApplicationAndStartJob

AWS CreateServerlessApplicationAndStartJob

Certified

Create EMR Serverless app and start job

Creates an EMR Serverless application then starts a job run immediately on it. Returns applicationId and jobRunId; does not wait for job completion. Supports SPARK or HIVE job drivers based on applicationType.

yaml
type: io.kestra.plugin.aws.emr.CreateServerlessApplicationAndStartJob

Create an EMR Serverless app and start a Spark job

yaml
id: create_and_run_emr_serverless_app
namespace: company.team

tasks:
  - id: create_and_run
    type: io.kestra.plugin.aws.emr.CreateServerlessApplicationAndStartJob
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-central-1"
    releaseLabel: "emr-7.0.0"
    applicationType: "SPARK"
    executionRoleArn: "arn:aws:iam::123456789012:role/EMRServerlessRole"
    jobName: "example-job"
    entryPoint: "s3://my-bucket/jobs/script.py"
Properties

Application type

Valid values SPARK or HIVE; determines which job driver is built.

Job entry point

For SPARK, S3 URI to the main file passed to spark-submit; for HIVE, the Hive query text.

Execution role ARN

IAM role assumed by EMR Serverless for the application.

Job name

Name reported for the started job run.

Release label

EMR Serverless release, e.g., emr-6.3.0 or emr-7.0.0.

Access Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

Enable compatibility mode

Use it to connect to S3 bucket with S3 compatible services that don't support the new transport client.

The endpoint with which the SDK should communicate

This property allows you to use a different S3 compatible storage backend.

Force path style access

Must only be used when compatibilityMode is enabled.

Reference (ref) of the pluginDefaults to apply to this task.

AWS region with which the SDK should communicate

Secret Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

The AWS STS endpoint with which the SDKClient should communicate

AWS STS Role

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

AWS STS External Id

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

DefaultPT15M

AWS STS Session duration

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

AWS STS Session name

This property is only used when an stsRoleArn is defined.

Application id

Job run id

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

OPA (Open Policy Agent)
OPA (Open Policy Agent)
Enterprise

Tasks to manage Open Policy Agent policies and decisions, including upload, evaluation, and partial evaluation.

Infrastructure

5 tasks

3 blueprints