JarSubmit
Submit a Spark job to a remote cluster using a JAR file.
type: "io.kestra.plugin.spark.JarSubmit"
id: spark_jar_submit
namespace: company.team
inputs:
- id: file
type: FILE
tasks:
- id: jar_submit
type: io.kestra.plugin.spark.JarSubmit
runner: DOCKER
master: spark://localhost:7077
mainResource: {{ inputs.file }}
mainClass: spark.samples.App
YES
The application class name for Java/Scala applications.
YES
The main application resource.
This should be the location of a JAR file for Scala/Java applications, or a Python script for PySpark applications. Must be an internal storage URI.
YES
Spark master hostname for the application.
Spark master URL formats.
YES
Adds a file to be submitted with the application.
Must be an internal storage URI.
YES
Command line arguments for the application.
YES
Configuration properties for the application.
YES
bitnami/spark
The task runner container image, only used if the task runner is container-based.
YES
CLIENT
CLUSTER
Deploy mode for the application.
NO
Deprecated, use 'taskRunner' instead
YES
Additional environment variables for the current process.
YES
Additional JAR files to be submitted with the application.
Must be an internal storage URI.
YES
Spark application name.
YES
PROCESS
DOCKER
Script runner to use.
Deprecated - use 'taskRunner' instead.
YES
spark-submit
The spark-submit
binary path.
NO
{
"type": "io.kestra.plugin.scripts.runner.docker.Docker"
}
The task runner to use.
Task runners are provided by plugins, each have their own properties.
YES
false
0
The exit code of the entire flow execution.
The output files' URIs in Kestra's internal storage.
The value extracted from the output of the executed commands
.
YES
YES
The maximum amount of kernel memory the container can use.
The minimum allowed value is 4MB
. Because kernel memory cannot be swapped out, a container which is starved of kernel memory may block host machine resources, which can have side effects on the host machine and on other containers. See the kernel-memory docs for more details.
YES
The maximum amount of memory resources the container can use.
Make sure to use the format number
+ unit
(regardless of the case) without any spaces.
The unit can be KB (kilobytes), MB (megabytes), GB (gigabytes), etc.
Given that it's case-insensitive, the following values are equivalent:
"512MB"
"512Mb"
"512mb"
"512000KB"
"0.5GB"
It is recommended that you allocate at least 6MB
.
YES
Allows you to specify a soft limit smaller than memory
which is activated when Docker detects contention or low memory on the host machine.
If you use memoryReservation
, it must be set lower than memory
for it to take precedence. Because it is a soft limit, it does not guarantee that the container doesn’t exceed the limit.
YES
The total amount of memory
and swap
that can be used by a container.
If memory
and memorySwap
are set to the same value, this prevents containers from using any swap. This is because memorySwap
includes both the physical memory and swap space, while memory
is only the amount of physical memory that can be used.
YES
A setting which controls the likelihood of the kernel to swap memory pages.
By default, the host kernel can swap out a percentage of anonymous pages used by a container. You can set memorySwappiness
to a value between 0 and 100 to tune this percentage.
YES
YES
1
Docker image to use.
YES
Docker configuration file.
Docker configuration file that can set access credentials to private container registries. Usually located in ~/.docker/config.json
.
NO
Limits the CPU usage to a given maximum threshold value.
By default, each container’s access to the host machine’s CPU cycles is unlimited. You can set various constraints to limit a given container’s access to the host machine’s CPU cycles.
YES
YES
Docker entrypoint to use.
YES
Extra hostname mappings to the container network interface configuration.
YES
Docker API URI.
NO
Limits memory usage to a given maximum threshold value.
Docker can enforce hard memory limits, which allow the container to use no more than a given amount of user or system memory, or soft limits, which allow the container to use as much memory as it needs unless certain conditions are met, such as when the kernel detects low memory or contention on the host machine. Some of these options have different effects when used alone or when more than one option is set.
YES
Docker network mode to use e.g. host
, none
, etc.
YES
YES
ALWAYS
IF_NOT_PRESENT
ALWAYS
NEVER
The image pull policy for a container image and the tag of the image, which affect when Docker attempts to pull (download) the specified image.
YES
Size of /dev/shm
in bytes.
The size must be greater than 0. If omitted, the system uses 64MB.
YES
User in the Docker container.
YES
List of volumes to mount.
Must be a valid mount expression as string, example : /home/user:/app
.
Volumes mount are disabled by default for security reasons; you must enable them on server configuration by setting kestra.tasks.scripts.docker.volume-enabled
to true
.
YES
The registry authentication.
The auth
field is a base64-encoded authentication string of username: password
or a token.
YES
The identity token.
YES
The registry password.
YES
The registry URL.
If not defined, the registry will be extracted from the image name.
YES
The registry token.
YES
The registry username.
YES
A list of capabilities; an OR list of AND lists of capabilities.
YES
YES
YES
YES
Driver-specific options, specified as key/value pairs.
These options are passed directly to the driver.