Python
type: "io.kestra.core.tasks.scripts.Python"
Execute a Python script
With this Python task, we can execute a full python script.
The task will create a fresh virtualenv
for every tasks and allow you to install some python package define in requirements
property.
By convention, you need to define at least a main.py
files in inputFiles
that will be the script used.
But you are also able to add as many script as you need in inputFiles
.
You can also add a pip.conf
in inputFiles
to customize the pip download of dependencies (like a private registry).
You can send outputs & metrics from your python script that can be used by others tasks. In order to help, we inject a python package directly on the working dir.Here is an example usage:
from kestra import Kestra
Kestra.outputs({'test': 'value', 'int': 2, 'bool': True, 'float': 3.65})
Kestra.counter('count', 1, {'tag1': 'i', 'tag2': 'win'})
Kestra.timer('timer1', lambda: time.sleep(1), {'tag1': 'i', 'tag2': 'lost'})
Kestra.timer('timer2', 2.12, {'tag1': 'i', 'tag2': 'destroy'})
# Examples
Execute a python script
id: "python"
type: "io.kestra.core.tasks.scripts.Python"
inputFiles:
data.json: |
{"status": "OK"}
main.py: |
from kestra import Kestra
import json
import requests
import sys
result = json.loads(open(sys.argv[1]).read())
print(f"python script {result['status']}")
response = requests.get('http://google.com')
print(response.status_code)
Kestra.outputs({'status': response.status_code, 'text': response.text})
pip.conf: |
# some specific pip repository configuration
args:
- data.json
requirements:
- requests
# Properties
# args
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
Python command args
Arguments list to pass to main python script
# commands
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ✔️
- Default:
[./bin/python main.py]
- Min items:
1
The commands to run
Default command will be launched with ./bin/python main.py
# dockerOptions
- Type: DockerOptions
- Dynamic: ❓
- Required: ❌
Docker options when using runner
DOCKER
# env
- Type: object
- SubType: string
- Dynamic: ✔️
- Required: ❌
Additional environnements variable to add for current process.
# exitOnFailed
- Type: boolean
- Dynamic: ❌
- Required: ✔️
- Default:
true
Exit if any non true return value
This tells bash that it should exit the script if any statement returns a non-true return value.
The benefit of using -e is that it prevents errors snowballing into serious issues when they could have been caught earlier.
# files
🔒 Deprecated
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
The list of files that will be uploaded to internal storage,
use outputFiles
property instead
# inputFiles
- Type: object
- SubType: string
- Dynamic: ✔️
- Required: ❌
Input files are extra files supplied by user that make it simpler organize code.
Describe a files map that will be written and usable in execution context. In python execution context is in a temp folder, for bash scripts, you can reach files using a workingDir variable like 'source /myfile.sh'
# interpreter
- Type: string
- Dynamic: ❌
- Required: ✔️
- Default:
/bin/sh
- Min length:
1
Interpreter to used
# interpreterArgs
- Type: array
- SubType: string
- Dynamic: ❌
- Required: ❌
- Default:
[-c]
Interpreter args used
# outputFiles
- Type: array
- SubType: string
- Dynamic: ❌
- Required: ❌
Output file list that will be uploaded to internal storage
List of key that will generate temporary files.
On the command, just can use with special variable named outputFiles.key
.
If you add a files with ["first"]
, you can use the special vars echo 1 >> {[ outputFiles.first }}
and you used on others tasks using {{ outputs.task-id.files.first }}
# outputsFiles
🔒 Deprecated
- Type: array
- SubType: string
- Dynamic: ❌
- Required: ❌
Deprecated Output file
use outputFiles
# pythonPath
- Type: string
- Dynamic: ✔️
- Required: ✔️
- Default:
python
- Min length:
1
The python interpreter to use
Set the python interpreter path to use
# requirements
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
Requirements are python dependencies to add to the python execution process
Python dependencies list to setup in the virtualenv, in the same format than requirements.txt
# runner
Type: string
Dynamic: ❌
Required: ✔️
Default:
PROCESS
Possible Values:
PROCESS
DOCKER
Runner to use
# warningOnStdErr
- Type: boolean
- Dynamic: ❌
- Required: ✔️
- Default:
true
Use
WARNING
state if any stdErr is sent
# Outputs
# exitCode
- Type: integer
- Default:
0
The exit code of the whole execution
# files
🔒 Deprecated
- Type: object
- SubType: string
Deprecated output files
use outputFiles
# outputFiles
- Type: object
- SubType: string
The output files uri in Kestra internal storage
# stdErrLineCount
- Type: integer
- Default:
0
The standard error line count
# stdOutLineCount
- Type: integer
- Default:
0
The standard output line count
# vars
- Type: object
The value extract from output of the commands
# Definitions
# DockerOptions
# dockerConfig
- Type: string
- Dynamic: ✔️
- Required: ❌
Docker config file
Full file that can be used to configure private registries, ...
# dockerHost
- Type: string
- Dynamic: ✔️
- Required: ❌
- Default:
unix:///var/run/docker.sock
Docker api uri
# entryPoint
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
Docker entrypoint to use
# extraHosts
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
Docker extra host to use
# image
- Type: string
- Dynamic: ✔️
- Required: ✔️
- Min length:
1
Docker image to use
# networkMode
- Type: string
- Dynamic: ✔️
- Required: ❌
Docker network mode to use
# user
- Type: string
- Dynamic: ✔️
- Required: ❌
Docker user to use
# volumes
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
List of volumes to mount
Must be a valid mount expression as string, example : /home/user:/app
Volumes mount are disabled by default for security reasons, you must enabled on server configuration with kestra.tasks.scripts.docker.volume-enabled
to true