Core Plugins and tasks LogShipper
CertifiedForward workflow execution logs to one or more desired destinations.
Core Plugins and tasks LogShipper
Forward workflow execution logs to one or more desired destinations.
The Log Shipper task extracts logs from the Kestra backend and loads them to desired destinations including Datadog, Elasticsearch, New Relic, OpenTelemetry, AWS CloudWatch, Google Operational Suite, and Azure Monitor.
The task works incrementally in batches:
- Determines the starting timestamp using either:
- The last successfully processed log's timestamp (persisted in KV Store using the
offsetKey) - Current time minus
lookbackPeriodduration if no previous state exists
- The last successfully processed log's timestamp (persisted in KV Store using the
- Sends retrieved logs through configured
logExporters - Stores the timestamp of the last processed log to maintain state between executions
- Subsequent runs continue from the last stored timestamp
This incremental approach ensures reliable log forwarding without gaps or duplicates.
type: io.kestra.plugin.ee.core.log.LogShipperExamples
Ship logs to multiple destinations
id: logShipper
namespace: system
tasks:
- id: shipLogs
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: logShipperOffset
delete: false
logExporters:
- id: file
type: io.kestra.plugin.ee.core.log.FileLogExporter
- id: awsCloudWatch
type: io.kestra.plugin.ee.aws.cloudwatch.LogExporter
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: us-east-1
logGroupName: kestra
logStreamName: production
- id: S3LogExporter
type: io.kestra.plugin.ee.aws.s3.LogExporter
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "{{ vars.region }}"
format: JSON
bucket: logbucket
logFilePrefix: kestra-log-file
maxLinesPerFile: 1000000
- id: googleOperationalSuite
type: io.kestra.plugin.ee.gcp.operationalsuite.LogExporter
projectId: my-gcp-project
- id: gcs
type: io.kestra.plugin.ee.gcp.gcs.LogExporter
projectId: myProjectId
format: JSON
maxLinesPerFile: 10000
bucket: my-bucket
logFilePrefix: kestra-log-file
chunk: 1000
- id: azureMonitor
type: io.kestra.plugin.ee.azure.monitor.LogExporter
endpoint: https://endpoint-host.ingest.monitor.azure.com
tenantId: "{{ secret('AZURE_TENANT_ID') }}"
clientId: "{{ secret('AZURE_CLIENT_ID') }}"
clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
ruleId: dcr-69f0b123041d4d6e9f2bf72aad0b62cf
streamName: kestraLogs
- id: azureBlobStorage
type: io.kestra.plugin.ee.azure.storage.LogExporter
endpoint: https://myblob.blob.core.windows.net/
tenantId: "{{ secret('AZURE_TENANT_ID') }}"
clientId: "{{ secret('AZURE_CLIENT_ID') }}"
clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
containerName: logs
format: JSON
logFilePrefix: kestra-log-file
maxLinesPerFile: 1000000
chunk: 1000
- id: datadog
type: io.kestra.plugin.ee.datadog.LogExporter
basePath: https://http-intake.logs.datadoghq.eu
apiKey: "{{ secret('DATADOG_API_KEY') }}"
- id: elasticsearch
type: io.kestra.plugin.ee.elasticsearch.LogExporter
indexName: kestra-logs
connection:
basicAuth:
password: "{{ secret('ES_PASSWORD') }}"
username: kestra_user
hosts:
- https://elastic.example.com:9200
- id: opensearch
type: io.kestra.plugin.ee.opensearch.LogExporter
indexName: kestra-logs
connection:
basicAuth:
password: "{{ secret('ES_PASSWORD') }}"
username: kestra_user
hosts:
- https://elastic.example.com:9200
- id: newRelic
type: io.kestra.plugin.ee.newrelic.LogExporter
basePath: https://log-api.newrelic.com
apiKey: "{{ secret('NEWRELIC_API_KEY') }}"
- id: openTelemetry
type: io.kestra.plugin.ee.opentelemetry.LogExporter
otlpEndpoint: http://otel-collector:4318/v1/logs
authorizationHeaderName: Authorization
authorizationHeaderValue: "Bearer {{ secret('OTEL_TOKEN') }}"
triggers:
- id: dailySchedule
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 0 * * *"
disabled: true
Properties
logExporters *RequiredNon-dynamic
1List of log shippers
The list of log shippers to use for sending logs
Ship logs to a file inside Kestra's internal storage.
This task is designed to be used when no external tool's log shipper satisfies requirements. You can ship logs to the internal storage and use any other Kestra task to send it to a remote location.
Ship logs to the internal storage
id: logShipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: logSync
type: io.kestra.plugin.ee.core.log.LogShipper
logExporters:
- id: file
type: io.kestra.plugin.ee.core.log.FileLogExporter
format: JSON
maxLinesPerFile: 100
^[a-zA-Z0-9][a-zA-Z0-9_-]*1IONIONJSONFormat of the exported files
This property defines the format of the exported files.
kestra-log-filePrefix of the log files
This property sets the prefix of the log files name. The full file name will be logFilePrefix-localDateTime.json/ion.
Maximum number of lines per file
This property specifies the maximum number of lines per log file.
Send execution logs to a Syslog server in CEF format
Formats each log record as a syslog-framed CEF (Common Event Format) message and ships it to a Syslog collector or SIEM over UDP, TCP, or TLS. Records are batched in chunk-sized groups (default 1000); the underlying connection is opened once per execution and reused across the whole batch.
Ship audit logs to a Syslog collector over UDP
id: audit_log_shipper_udp
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: audit_log_export
type: io.kestra.plugin.ee.core.log.AuditLogShipper
resources:
- FLOW
- EXECUTION
lookbackPeriod: P1D
offsetKey: syslog-audit-offset
logExporters:
- id: syslogExporter
type: io.kestra.plugin.ee.syslog.LogExporter
host: siem.company.com
port: 514
protocol: UDP
Ship execution logs to a Syslog collector over TCP with TLS
id: log_shipper_tls
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
offsetKey: syslog-log-offset
logExporters:
- id: syslogExporter
type: io.kestra.plugin.ee.syslog.LogExporter
host: siem.company.com
port: 6514
protocol: TLS
trustStorePath: /etc/kestra/certs/siem-truststore.p12
trustStorePassword: "{{ secret('SYSLOG_TRUSTSTORE_PASSWORD') }}"
Syslog server host
Hostname or IP address of the Syslog collector, e.g. siem.company.com.
^[a-zA-Z0-9][a-zA-Z0-9_-]*1kestraApplication name
Value used for the syslog header APP-NAME field; defaults to kestra.
1000Chunk size
Number of log records buffered per network write; must be between 1 and 10000. Defaults to 1000.
PT10SConnection timeout
Maximum time to wait when establishing the TCP/TLS connection to the Syslog server, and the read timeout applied afterwards (covers the TLS handshake and any protocol-level reads). A collector that refuses the connection or stalls the handshake fails within this bound; it does not bound a blocked write to a collector that stopped draining data. Not used for UDP, which is fire-and-forget. Must be positive; a non-positive value (e.g. PT0S) falls back to the default. Defaults to PT10S (10 seconds).
KestraDevice product
Value used for the CEF Device Product header field; defaults to Kestra.
KestraDevice vendor
Value used for the CEF Device Vendor header field; defaults to Kestra.
Device version
Value used for the CEF Device Version header field; empty by default.
USERKERNUSERMAILDAEMONAUTHSYSLOGLPRNEWSUUCPCRONAUTHPRIVFTPNTPAUDITALERTCLOCKLOCAL0LOCAL1LOCAL2LOCAL3LOCAL4LOCAL5LOCAL6LOCAL7Syslog facility
Syslog facility used to compute the message PRI header value, e.g. USER, LOCAL0..LOCAL7, DAEMON. Defaults to USER.
Hostname
Value used for the syslog header HOSTNAME field; defaults to the worker's local hostname.
Syslog server port
TCP/UDP port of the Syslog collector; must be between 1 and 65535. Defaults to 514 for UDP and TCP, and 6514 for TLS.
TCPUDPTCPTLSTransport protocol
Transport used to deliver syslog messages: UDP (fire-and-forget datagrams), TCP (RFC 5425 octet-counting framing over a plain socket), or TLS (same framing over an encrypted socket). Defaults to TCP.
falseSkip TLS certificate verification
When true, disables all TLS certificate verification for the Syslog connection. This is unsafe and should only be used for testing against a server with a self-signed certificate that you cannot add to a truststore. Never enable this in production. Defaults to false.
Truststore password
Password protecting the truststore referenced by trustStorePath, if any.
Truststore path
Path to a PKCS12/JKS truststore file, readable from the worker's filesystem, containing the CA certificate(s) needed to trust the Syslog server's TLS certificate. Only used when protocol is TLS; falls back to the JVM default trust store when unset.
PKCS12Truststore type
Type of the truststore file referenced by trustStorePath; defaults to PKCS12.
Send logs to an OTLP collector
Exports Kestra log records to an OTLP/HTTP collector in batches; requires otlpEndpoint, can add authentication headers, batches chunk logs per request (default 1000), and emits counters for requests and logs plus a duration timer.
Ship logs using OTLP
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: OTLPLogExporter
type: io.kestra.plugin.ee.opentelemetry.LogExporter
otlpEndpoint: http://localhost:4318/v1/logs
authorizationHeaderName: Authorization
authorizationHeaderValue: Bearer token
^[a-zA-Z0-9][a-zA-Z0-9_-]*1OTLP endpoint
HTTP endpoint for the OTLP logs API
Authentication header name
Header name used for authentication when calling the collector
Authentication header value
Header value paired with the authentication header name
1000The chunk size for every bulk request
Number of log records per export batch; default 1000
Export logs to AWS CloudWatch
This task is designed to send logs to AWS CloudWatch.
Ship logs to AWS
id: log_shipper
namespace: system
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: AWSLogExporter
type: io.kestra.plugin.ee.aws.cloudwatch.LogExporter
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "{{ vars.region }}"
logGroupName: group_name
logStreamName: stream_name
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
^[a-zA-Z0-9][a-zA-Z0-9_-]*1The name of the log group
The name of the log stream
AWS region with which the SDK should communicate
Access Key Id in order to connect to AWS
If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
1000The chunk size for every bulk request
The endpoint with which the SDK should communicate
This property allows you to use a different S3 compatible storage backend.
Secret Key Id in order to connect to AWS
If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource
If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
The AWS STS endpoint with which the SDKClient should communicate
AWS STS Role
The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
AWS STS External Id
A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.
PT15MAWS STS Session duration
The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.
AWS STS Session name
This property is only used when an stsRoleArn is defined.
Export logs to S3
This task is designed to send logs to S3.
Ship logs to AWS S3
id: log_shipper
namespace: system
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: S3LogExporter
type: io.kestra.plugin.ee.aws.s3.LogExporter
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "{{ vars.region }}"
format: JSON
bucket: logbucket
logFilePrefix: kestra-log-file
maxLinesPerFile: 1000000
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
S3 Bucket to upload logs files
The bucket where log files are going to be imported
^[a-zA-Z0-9][a-zA-Z0-9_-]*1AWS region with which the SDK should communicate
Access Key Id in order to connect to AWS
If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
1000The chunk size for every bulk request
The endpoint with which the SDK should communicate
This property allows you to use a different S3 compatible storage backend.
JSONIONJSONFormat of the exported files
The format of the exported files
kestra-log-filePrefix of the log files
The prefix of the log files name. The full file name will be logFilePrefix-localDateTime.json/ion
100000Maximum number of lines per file
The maximum number of lines per file
Secret Key Id in order to connect to AWS
If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource
If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
The AWS STS endpoint with which the SDKClient should communicate
AWS STS Role
The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
AWS STS External Id
A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.
PT15MAWS STS Session duration
The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.
AWS STS Session name
This property is only used when an stsRoleArn is defined.
Bulk export logs to Elasticsearch
Streams execution logs to an Elasticsearch index via the Bulk API. Uses rendered indexName and chunk for batching (default 1000 records per bulk request); document IDs are derived from log timestamps, so ensure index mappings accept the task log schema.
Ship logs to Elasticsearch
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: logSync
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: ElasticsearchLogExporter
type: io.kestra.plugin.ee.elasticsearch.LogExporter
connection:
hosts:
- "http://localhost:9200/"
indexName: "logs"
Elasticsearch connection settings
Hosts, authentication, headers, and TLS options reused for all requests.
io.kestra.plugin.ee.elasticsearch.ElasticsearchConnection
1Elasticsearch hosts
HTTP/HTTPS endpoints with scheme and port, e.g. https://elasticsearch.com: 9200
Basic auth configuration
io.kestra.plugin.ee.elasticsearch.ElasticsearchConnection-BasicAuth
Basic auth password
Basic auth username
Extra HTTP headers
Rendered Name: Value pairs sent on every request, e.g. Authorization: Token XYZ
Request path prefix
Prefixes every endpoint (e.g. /my/base) for clusters behind a proxy that requires a base path
Fail on deprecation warnings
If true, any response with Elasticsearch warning headers is treated as an error
8Target Elasticsearch server major version
Major version used for compatibility headers (Accept and Content-Type). Set to 8 for Elasticsearch 8 clusters or 9 for Elasticsearch 9 clusters.
Trust all SSL certificates
WARNING — SECURITY RISK: When enabled, disables BOTH TLS certificate validation (TrustAllStrategy) AND hostname verification (NoopHostnameVerifier). This makes the connection vulnerable to man-in-the-middle attacks and must never be used in production. Prefer configuring a proper truststore for self-signed certificates instead.
^[a-zA-Z0-9][a-zA-Z0-9_-]*1Target index name
Rendered per execution; index must accept the task log schema.
1000Bulk chunk size
Number of log documents per bulk request (default 1000); tune for throughput versus memory.
Send flow logs to Datadog Logs
Pushes task logs to Datadog Logs intake v2 over HTTPS. Batches records (default 1000) and sends them with the DD-API-KEY header against the configured region endpoint.
Ship logs to the internal storage
id: logShipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: DatadogLogExporter
type: io.kestra.plugin.ee.datadog.LogExporter
basePath: https://http-intake.logs.datadoghq.eu
apiKey: "{{ secret('DATADOG_API_KEY') }}"
Datadog API key
API key sent in the DD-API-KEY header for the target account
Datadog logs intake URL
Base URL of the Datadog region (e.g., https://http-intake.logs.datadoghq.com); the task appends /api/v2/logs
^[a-zA-Z0-9][a-zA-Z0-9_-]*11000Batch size
Number of log entries per request; default 1000
HTTP client options
Optional HTTP client overrides such as timeouts, proxy, or TLS
io.kestra.core.http.client.configurations.HttpConfiguration
falseIf true, allow a failed response code (response code >= 400)
List of response code allowed for this request
The authentication to use.
io.kestra.core.http.client.configurations.BasicAuthConfiguration
The password for HTTP basic authentication.
The username for HTTP basic authentication.
io.kestra.core.http.client.configurations.BearerAuthConfiguration
The token for bearer token authentication.
io.kestra.core.http.client.configurations.DigestAuthConfiguration
The password for HTTP Digest authentication.
The username for HTTP Digest authentication.
The password for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.
The username for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.
durationThe time allowed to establish a connection to the server before failing.
durationThe time an idle connection can remain in the client's connection pool before being closed.
UTF-8The default charset for the request.
java.nio.charset.Charset
trueWhether to enable TCP Keep-Alive extended socket options (TCP_KEEPIDLE, TCP_KEEPINTERVAL, TCP_KEEPCOUNT).
Set to false when running on Windows workers, as these extended socket options are not supported by the Windows JDK and will cause connection failures.
trueWhether redirects should be followed automatically.
ALLTRACEDEBUGINFOWARNERROROFFNOT_SPECIFIEDThe log level for the HTTP client.
REQUEST_HEADERSREQUEST_BODYRESPONSE_HEADERSRESPONSE_BODYThe enabled log.
The maximum content length of the response.
The proxy configuration.
io.kestra.core.http.client.configurations.ProxyConfiguration
The address of the proxy server.
The password for proxy authentication.
The port of the proxy server.
DIRECTDIRECTHTTPSOCKSThe type of proxy to use.
The username for proxy authentication.
The address of the proxy server.
The password for proxy authentication.
The port of the proxy server.
DIRECTHTTPSOCKSThe type of proxy to use.
The username for proxy authentication.
durationThe time allowed for a read connection to remain idle before closing it.
durationThe maximum time allowed for reading data from the server before failing.
The SSL request options
io.kestra.core.http.client.configurations.SslOptions
Whether to disable checking of the remote SSL certificate.
Only applies if no trust store is configured. Note: This makes the SSL connection insecure and should only be used for testing. If you are using a self-signed certificate, set up a trust store instead.
The timeout configuration.
io.kestra.core.http.client.configurations.TimeoutConfiguration
The time allowed to establish a connection to the server before failing.
PT5MThe time allowed for a read connection to remain idle before closing it.
LogExporterService name
Sets Datadog service; defaults to LogExporter
KestraLog source
Sets Datadog ddsource; defaults to Kestra
Export execution logs to OpenSearch
Streams Kestra log records into an OpenSearch index using bulk requests; control batch size and connection settings, index name is required.
Ship logs to Opensearch
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: logSync
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: OpensearchLogExporter
type: io.kestra.plugin.ee.opensearch.LogExporter
connection:
hosts:
- "http://localhost:9200/"
indexName: "logs"
OpenSearch connection
Shared HTTP settings including hosts, auth, headers, path prefix, and TLS options.
io.kestra.plugin.ee.opensearch.OpensearchConnection
Target OpenSearch HTTP endpoints
One or more full URLs with scheme and port, e.g. https://opensearch.com: 9200.
Authenticate with HTTP basic auth
Provide username (and optionally password) to send Basic credentials with every request.
io.kestra.plugin.ee.opensearch.OpensearchConnection-BasicAuth
Basic auth password
Rendered before use; leave empty for anonymous access.
Basic auth username
Rendered before use; required when password is provided.
Send custom HTTP headers
Each entry must be Name: Value, e.g. Authorization: Token XYZ; rendered before being sent on every request.
Prefix all request paths
Prepends this path (e.g. /my/base) to every endpoint; use when OpenSearch sits behind a proxy that requires a base path.
Fail on deprecation warnings
When true, any response carrying a warning header is treated as a failure; disabled unless set.
Trust all SSL certificates
Ignore certificate chain validation (insecure); useful only for self-signed test clusters. Hostname verification is still enforced. WARNING: enabling this option exposes the connection to man-in-the-middle attacks and must never be used in production environments or when connecting to external or sensitive data stores.
^[a-zA-Z0-9][a-zA-Z0-9_-]*1Target index name
Rendered before use; must exist or auto-create per cluster settings.
1000Bulk chunk size
Number of log records per bulk request; defaults to 1000.
Send execution logs to Graylog
Sends Flow logs to a Graylog GELF HTTP input over HTTP. Batches records in chunk-sized groups (default 1000) and sets the GELF host field from graylogHost (default Kestra).
Ship logs to Graylog
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
logExporters:
- id: GraylogExporter
type: io.kestra.plugin.ee.graylog.LogExporter
endpoint: "http://localhost:12201/gelf"
graylogHost: "Kestra"
chunk: 1000
Graylog GELF HTTP endpoint
URL of the GELF HTTP input, e.g. http://graylog: 12201/gelf
^[a-zA-Z0-9][a-zA-Z0-9_-]*11000Chunk size per batch
Number of log records sent per HTTP batch; default 1000
KestraGraylog host field
Value used for the GELF host field; defaults to Kestra when not provided
HTTP client configuration
Optional HTTP client settings (timeouts, authentication, headers) applied to Graylog requests
io.kestra.core.http.client.configurations.HttpConfiguration
falseIf true, allow a failed response code (response code >= 400)
List of response code allowed for this request
The authentication to use.
io.kestra.core.http.client.configurations.BasicAuthConfiguration
The password for HTTP basic authentication.
The username for HTTP basic authentication.
io.kestra.core.http.client.configurations.BearerAuthConfiguration
The token for bearer token authentication.
io.kestra.core.http.client.configurations.DigestAuthConfiguration
The password for HTTP Digest authentication.
The username for HTTP Digest authentication.
The password for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.
The username for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.
durationThe time allowed to establish a connection to the server before failing.
durationThe time an idle connection can remain in the client's connection pool before being closed.
UTF-8The default charset for the request.
java.nio.charset.Charset
trueWhether to enable TCP Keep-Alive extended socket options (TCP_KEEPIDLE, TCP_KEEPINTERVAL, TCP_KEEPCOUNT).
Set to false when running on Windows workers, as these extended socket options are not supported by the Windows JDK and will cause connection failures.
trueWhether redirects should be followed automatically.
ALLTRACEDEBUGINFOWARNERROROFFNOT_SPECIFIEDThe log level for the HTTP client.
REQUEST_HEADERSREQUEST_BODYRESPONSE_HEADERSRESPONSE_BODYThe enabled log.
The maximum content length of the response.
The proxy configuration.
io.kestra.core.http.client.configurations.ProxyConfiguration
The address of the proxy server.
The password for proxy authentication.
The port of the proxy server.
DIRECTDIRECTHTTPSOCKSThe type of proxy to use.
The username for proxy authentication.
The address of the proxy server.
The password for proxy authentication.
The port of the proxy server.
DIRECTHTTPSOCKSThe type of proxy to use.
The username for proxy authentication.
durationThe time allowed for a read connection to remain idle before closing it.
durationThe maximum time allowed for reading data from the server before failing.
The SSL request options
io.kestra.core.http.client.configurations.SslOptions
Whether to disable checking of the remote SSL certificate.
Only applies if no trust store is configured. Note: This makes the SSL connection insecure and should only be used for testing. If you are using a self-signed certificate, set up a trust store instead.
The timeout configuration.
io.kestra.core.http.client.configurations.TimeoutConfiguration
The time allowed to establish a connection to the server before failing.
PT5MThe time allowed for a read connection to remain idle before closing it.
Send flow logs to Dash0
Exports Kestra log records to Dash0 via OTLP/HTTP. Authenticates with a Bearer token and optionally routes to a specific dataset. Batches chunk logs per request (default 1000) and emits counters for requests and logs plus a duration timer.
Ship logs to Dash0
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: dash0LogExporter
type: io.kestra.plugin.ee.dash0.LogExporter
endpoint: https://ingress.eu-west-1.aws.dash0.com/v1/logs
authToken: "{{ secret('DASH0_AUTH_TOKEN') }}"
dataset: my-dataset
Dash0 auth token
Bearer token sent in the Authorization header
Dash0 OTLP ingestion URL
Full OTLP/HTTP endpoint for Dash0 logs (e.g. https://ingress.eu-west-1.aws.dash0.com/v1/logs)
^[a-zA-Z0-9][a-zA-Z0-9_-]*11000Batch size
Number of log records per export batch; default 1000
defaultDash0 dataset
Value for the Dash0-Dataset header; defaults to default
Send logs to Splunk HEC
Sends log records to a Splunk HTTP Event Collector endpoint using buffered POST requests.
Defaults to chunking 1,000 events per request and sets source to Kestra; requires a HEC token in the Authorization header.
Ship logs to Splunk
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: SplunkLogExporter
type: io.kestra.plugin.ee.splunk.LogExporter
host: YOUR_HOST
token: "{{ secret('SPLUNK_TOKEN') }}"
Splunk HEC URL
Base URL of the Splunk HTTP Event Collector endpoint (e.g. https://host: 8088); the task appends /services/collector
^[a-zA-Z0-9][a-zA-Z0-9_-]*1Splunk HEC token
Token sent in the Authorization header to authenticate with Splunk; store it as a secret
1000Batch size
Number of log events buffered per POST request; defaults to 1000
HTTP client options
Optional HTTP client configuration such as timeouts, TLS, or proxy settings
io.kestra.core.http.client.configurations.HttpConfiguration
falseIf true, allow a failed response code (response code >= 400)
List of response code allowed for this request
The authentication to use.
io.kestra.core.http.client.configurations.BasicAuthConfiguration
The password for HTTP basic authentication.
The username for HTTP basic authentication.
io.kestra.core.http.client.configurations.BearerAuthConfiguration
The token for bearer token authentication.
io.kestra.core.http.client.configurations.DigestAuthConfiguration
The password for HTTP Digest authentication.
The username for HTTP Digest authentication.
The password for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.
The username for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.
durationThe time allowed to establish a connection to the server before failing.
durationThe time an idle connection can remain in the client's connection pool before being closed.
UTF-8The default charset for the request.
java.nio.charset.Charset
trueWhether to enable TCP Keep-Alive extended socket options (TCP_KEEPIDLE, TCP_KEEPINTERVAL, TCP_KEEPCOUNT).
Set to false when running on Windows workers, as these extended socket options are not supported by the Windows JDK and will cause connection failures.
trueWhether redirects should be followed automatically.
ALLTRACEDEBUGINFOWARNERROROFFNOT_SPECIFIEDThe log level for the HTTP client.
REQUEST_HEADERSREQUEST_BODYRESPONSE_HEADERSRESPONSE_BODYThe enabled log.
The maximum content length of the response.
The proxy configuration.
io.kestra.core.http.client.configurations.ProxyConfiguration
The address of the proxy server.
The password for proxy authentication.
The port of the proxy server.
DIRECTDIRECTHTTPSOCKSThe type of proxy to use.
The username for proxy authentication.
The address of the proxy server.
The password for proxy authentication.
The port of the proxy server.
DIRECTHTTPSOCKSThe type of proxy to use.
The username for proxy authentication.
durationThe time allowed for a read connection to remain idle before closing it.
durationThe maximum time allowed for reading data from the server before failing.
The SSL request options
io.kestra.core.http.client.configurations.SslOptions
Whether to disable checking of the remote SSL certificate.
Only applies if no trust store is configured. Note: This makes the SSL connection insecure and should only be used for testing. If you are using a self-signed certificate, set up a trust store instead.
The timeout configuration.
io.kestra.core.http.client.configurations.TimeoutConfiguration
The time allowed to establish a connection to the server before failing.
PT5MThe time allowed for a read connection to remain idle before closing it.
KestraLog source
Value for the Splunk source field; defaults to Kestra
Send logs to New Relic Logs API
Sends buffered Kestra log records to New Relic's /log/v1 endpoint over HTTPS, batching up to chunk (default 1000) records per request. Requires an Api-Key (license or ingest key); HTTP timeouts, retries, and proxies can be tuned via options.
Ship logs to Elasticsearch
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: NewRelicLogExporter
type: io.kestra.plugin.ee.newrelic.LogExporter
basePath: https://log-api.newrelic.com
API or License key
Key sent as Api-Key header for log ingest
Log ingest base URL
HTTPS endpoint for New Relic log ingest, e.g. https://log-api.newrelic.com
^[a-zA-Z0-9][a-zA-Z0-9_-]*11000Log batch size
Maximum records per request; defaults to 1000
HTTP client options
Optional HTTP settings (timeouts, retries, proxies) for calls to New Relic
io.kestra.core.http.client.configurations.HttpConfiguration
falseIf true, allow a failed response code (response code >= 400)
List of response code allowed for this request
The authentication to use.
io.kestra.core.http.client.configurations.BasicAuthConfiguration
The password for HTTP basic authentication.
The username for HTTP basic authentication.
io.kestra.core.http.client.configurations.BearerAuthConfiguration
The token for bearer token authentication.
io.kestra.core.http.client.configurations.DigestAuthConfiguration
The password for HTTP Digest authentication.
The username for HTTP Digest authentication.
The password for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.
The username for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.
durationThe time allowed to establish a connection to the server before failing.
durationThe time an idle connection can remain in the client's connection pool before being closed.
UTF-8The default charset for the request.
java.nio.charset.Charset
trueWhether to enable TCP Keep-Alive extended socket options (TCP_KEEPIDLE, TCP_KEEPINTERVAL, TCP_KEEPCOUNT).
Set to false when running on Windows workers, as these extended socket options are not supported by the Windows JDK and will cause connection failures.
trueWhether redirects should be followed automatically.
ALLTRACEDEBUGINFOWARNERROROFFNOT_SPECIFIEDThe log level for the HTTP client.
REQUEST_HEADERSREQUEST_BODYRESPONSE_HEADERSRESPONSE_BODYThe enabled log.
The maximum content length of the response.
The proxy configuration.
io.kestra.core.http.client.configurations.ProxyConfiguration
The address of the proxy server.
The password for proxy authentication.
The port of the proxy server.
DIRECTDIRECTHTTPSOCKSThe type of proxy to use.
The username for proxy authentication.
The address of the proxy server.
The password for proxy authentication.
The port of the proxy server.
DIRECTHTTPSOCKSThe type of proxy to use.
The username for proxy authentication.
durationThe time allowed for a read connection to remain idle before closing it.
durationThe maximum time allowed for reading data from the server before failing.
The SSL request options
io.kestra.core.http.client.configurations.SslOptions
Whether to disable checking of the remote SSL certificate.
Only applies if no trust store is configured. Note: This makes the SSL connection insecure and should only be used for testing. If you are using a self-signed certificate, set up a trust store instead.
The timeout configuration.
io.kestra.core.http.client.configurations.TimeoutConfiguration
The time allowed to establish a connection to the server before failing.
PT5MThe time allowed for a read connection to remain idle before closing it.
Export logs to Azure Monitor
This task is designed to send logs to Azure Monitor.
Ship logs to Azure Monitor
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: AzureLogExporter
type: io.kestra.plugin.ee.azure.monitor.LogExporter
endpoint: https://endpoint-host.ingest.monitor.azure.com
tenantId: tenant_id
clientId: client_id
clientSecret: client_secret
ruleId: dcr-69f0b123041d4d6e9f2bf72aad0b62cf
streamName: Custom-JSONLogs
Url of the Data Collection Endpoint
^[a-zA-Z0-9][a-zA-Z0-9_-]*1Id of the Data Collection Rule
Name of the stream
1000The chunk size for every bulk request
Client ID
Client ID of the Azure service principal. If you don't have a service principal, refer to create a service principal with Azure CLI.
Client Secret
Service principal client secret. The tenantId, clientId and clientSecret of the service principal are required for this credential to acquire an access token.
PEM Certificate
Your stored PEM certificate.
The tenantId, clientId and clientCertificate of the service principal are required for this credential to acquire an access token.
Tenant ID
Export logs to Azure Blob Storage
This task is designed to send logs to Azure Blob Storage.
Ship logs to Azure Blob Storage
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: AzureLogExporter
type: io.kestra.plugin.ee.azure.storage.LogExporter
endpoint: https://myblob.blob.core.windows.net/
tenantId: tenant_id
clientId: client_id
clientSecret: client_secret
containerName: logs
format: JSON
logFilePrefix: kestra-log-file
maxLinesPerFile: 1000000
chunk: 1000
Name of the container
Name of the container in the blob storage
Url of the Blob Storage
^[a-zA-Z0-9][a-zA-Z0-9_-]*11000The chunk size for every bulk request
Client ID
Client ID of the Azure service principal. If you don't have a service principal, refer to create a service principal with Azure CLI.
Client Secret
Service principal client secret. The tenantId, clientId and clientSecret of the service principal are required for this credential to acquire an access token.
Connection string of the Storage Account
JSONIONJSONFormat of the exported files
The format of the exported files
kestra-log-filePrefix of the log files
The prefix of the log files name. The full file name will be logFilePrefix-localDateTime.json/ion
100000Maximum number of lines per file
The maximum number of lines per file
PEM Certificate
Your stored PEM certificate.
The tenantId, clientId and clientCertificate of the service principal are required for this credential to acquire an access token.
The SAS token to use for authenticating requests
This string should only be the query parameters (with or without a leading '?') and not a full URL.
Shared Key access key for authenticating requests
Shared Key account name for authenticating requests
Tenant ID
Write logs to Google Cloud Storage
Streams Kestra log records into GCS objects in JSON or ION, splitting files after a configurable line count (default 100,000) and buffering writes in chunks (default 1000). Uses service account credentials or impersonation; requires bucket write permission and emits request/file/log metrics.
Ship logs to GCP
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
lookbackPeriod: P1D
logExporters:
- id: GCPLogExporter
type: io.kestra.plugin.ee.gcp.gcs.LogExporter
projectId: myProjectId
format: JSON
maxLinesPerFile:10000
bucket: my-bucket
logFilePrefix: kestra-log-file
chunk: 1000
Target GCS bucket
Existing bucket receiving the log objects; caller must have write access.
^[a-zA-Z0-9][a-zA-Z0-9_-]*11000Batch size per upload request
Number of log lines buffered before writing to GCS; defaults to 1000. Lower to reduce memory or request payload.
JSONIONJSONFile serialization format
Choose JSON or ION output; defaults to JSON.
The GCP service account to impersonate
Service account email to impersonate for API calls.
For Cloud Run runner, this value applies to API calls used to create and run the job (--impersonate-service-account equivalent).
It does not set the job execution identity (--service-account).
kestra-log-fileLog object name prefix
Prefix applied to each object name; final name is prefix-<timestamp>.json|ion. Defaults to kestra-log-file.
100000Maximum lines per file
Rotates to a new object after this many log lines; defaults to 100,000.
The GCP project ID
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
The GCP service account key
Service account JSON key used to authenticate API calls.
For Cloud Run runner job execution identity, this value is used as a fallback for --service-account
when runtimeServiceAccount is not provided.
Send logs to Google Cloud Logging
Streams Kestra log records to Google Cloud Logging (Operational Suite) using a service account or impersonation. Batches entries per request (default 1000) and publishes request/log metrics; requires project-level Logging Writer access.
Ship logs to GCP
id: log_shipper
namespace: company.team
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
tasks:
- id: log_export
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: GCPLogExporter
type: io.kestra.plugin.ee.gcp.operationalsuite.LogExporter
projectId: myProjectId
^[a-zA-Z0-9][a-zA-Z0-9_-]*11000Batch size per write request
Number of log entries sent in each Cloud Logging write call; defaults to 1000. Lower if you hit payload limits or latency spikes.
The GCP service account to impersonate
Service account email to impersonate for API calls.
For Cloud Run runner, this value applies to API calls used to create and run the job (--impersonate-service-account equivalent).
It does not set the job execution identity (--service-account).
The GCP project ID
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
The GCP service account key
Service account JSON key used to authenticate API calls.
For Cloud Run runner job execution identity, this value is used as a fallback for --service-account
when runtimeServiceAccount is not provided.
Export Kestra execution logs to Huawei Cloud Log Tank Service (LTS).
Sends execution logs in batches to an LTS log stream. Each log record is serialized as JSON
and posted to the LTS log-ingestion endpoint
POST /v2/{project_id}/lts/groups/{logGroupId}/streams/{logStreamId}/tenant/contents/high-accuracy
on the regional lts-access host (port 8102). The high-accuracy variant preserves each
record's original nanosecond timestamp instead of stamping the whole batch with one time.
Note: the lts-access ingestion endpoint (port 8102) is reachable from hosts inside the
target Huawei Cloud region (or via a jump server); it is not exposed on the public
management gateway (port 443).
Authentication supports static AK/SK, pre-obtained security tokens, and inline IAM STS
credential exchange via temporaryCredentials. The credential exchange runs once per
sendLogs invocation; for very long log streams that outlive the STS durationSeconds,
use long-lived AK/SK credentials or renew externally.
Ship Kestra execution logs to Huawei Cloud LTS daily
id: lts_log_shipper
namespace: company.team
tasks:
- id: ship_logs
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 1000
lookbackPeriod: P1D
logExporters:
- id: huawei_lts
type: io.kestra.plugin.ee.huawei.lts.LogExporter
region: eu-west-101
projectId: "{{ vars.project_id }}"
accessKeyId: "{{ secret('HUAWEI_ACCESS_KEY_ID') }}"
secretAccessKey: "{{ secret('HUAWEI_SECRET_ACCESS_KEY') }}"
logGroupId: "{{ vars.log_group_id }}"
logStreamId: "{{ vars.log_stream_id }}"
triggers:
- id: daily
type: io.kestra.plugin.core.trigger.Schedule
cron: "@daily"
^[a-zA-Z0-9][a-zA-Z0-9_-]*1LTS log group ID.
The UUID of the LTS log group to ingest into. Found in the LTS console under Log Groups.
LTS log stream ID.
The UUID of the LTS log stream within logGroupId. Found in the LTS console under Log Streams.
Huawei Cloud Project ID.
Identifies the region-scoped project. Required to build the LTS ingest URL.
Huawei Cloud region.
Region identifier such as eu-west-101, ap-southeast-1, or cn-north-4.
Access Key (AK) used to authenticate with Huawei Cloud.
Huawei Cloud access key used together with secretAccessKey to sign API requests.
Required for AK/SK-based authentication; not required when using temporaryCredentials.
Sensitive — always provide via {{ secret('NAME') }}.
1000Number of log records per LTS ingest request.
Defaults to 1000. Reduce if individual log records are very large.
Huawei Cloud Account Domain ID.
Required only when using temporaryCredentials with domain-scoped token scope.
Override the LTS service endpoint URL.
Replaces the region-derived endpoint (e.g. https://lts-access.eu-west-101.myhuaweicloud.eu).
Intended for testing against a local WireMock server; do not set in production.
Secret Key (SK) used to authenticate with Huawei Cloud.
Huawei Cloud secret key paired with accessKeyId.
Sensitive — always provide via {{ secret('NAME') }}.
Pre-obtained Huawei Cloud security token.
When set, requests are authenticated with this token in the X-Auth-Token header
instead of AK/SK signing. Sensitive.
Inline IAM credential exchange.
When set, the connection layer calls the Huawei IAM STS API once per task execution and
uses the returned temporary AK/SK + security token. The exchange runs once at invocation
start; credentials will expire mid-run for streams that outlive durationSeconds.
io.kestra.plugin.huawei.TemporaryCredentialsConfig
PASSWORDPASSWORDTOKENAuthentication method.
Controls which credentials are used to obtain the session token before exchanging for temporary STS credentials.
PASSWORD(default): provideusername,password, anddomainName.TOKEN: provide an existingiamToken(X-Auth-Token).
Account domain name (PASSWORD method only).
The Huawei Cloud account name (domain name) that owns the IAM user.
Required when authMethod is PASSWORD. Visible in the Huawei Cloud console under
My Credentials → Domain Name.
900Lifetime of the temporary credentials in seconds.
How long the returned temporary AK/SK/security-token should remain valid. Huawei Cloud accepts values between 900 (15 minutes) and 86400 (24 hours). Defaults to 900 seconds.
myhuaweicloud.comHuawei Cloud IAM endpoint suffix.
Domain suffix used to build the IAM endpoint URL when no explicit endpoint override is set.
Defaults to myhuaweicloud.com. Set to myhuaweicloud.eu for the European sovereign cloud
(region eu-west-101 / EU-Dublin).
IAM token to exchange (TOKEN method only).
An existing Huawei Cloud X-Auth-Token to exchange for temporary STS credentials.
Required when authMethod is TOKEN. Sensitive — always provide via {{ secret('NAME') }}.
IAM password (PASSWORD method only).
Password for the IAM user identified by username.
Required when authMethod is PASSWORD.
Sensitive — always provide via {{ secret('NAME') }}.
Project name for project-scoped tokens (PASSWORD method only).
Overrides the project name used for scope=PROJECT token requests.
Defaults to the task's region value when omitted, which is correct for most regions.
PROJECTPROJECTDOMAINToken scope (PASSWORD method only).
Scope of the session token obtained during password authentication.
PROJECT(default): token is scoped to the project matchingprojectName(or the task'sregionwhenprojectNameis omitted). Use for most downstream tasks.DOMAIN: token is scoped to the domain.
IAM username (PASSWORD method only).
Huawei Cloud IAM username. Required when authMethod is PASSWORD.
delete booleanstring
Delete logs after export
The log shipper will delete the exported logs
executionId string
Execution to search
The execution ID to use to filter logs
flowId string
Flow to search
The flow ID to use to filter logs
logLevelFilter string
INFOLog level to send
This property specifies the minimum log level to send.
lookbackPeriod string
P1DStarting duration before now
If no previous execution or state exists, the fetch start date is set to the current time minus this duration
maximumMessageSize integerstring
Maximum size of messages in logs
Set the maximum size of the message inside logs (in number of characters). If set, it will truncate messages that are longer than the maximum message size.
namespace string
Namespace to search
The namespace to use to filter logs
offsetKey string
Prefix of the KVStore key
The prefix of the KVStore key that contains the last execution's end fetched date
pluginDefaultsRef Non-dynamicstring
Reference (ref) of the pluginDefaults to apply to this task.
Outputs
endFetchedDate string
date-timeThe zoned date-time of the last fetched log, used as the starting date for the next execution
outputs object
The outputs generated by each log shipper
io.kestra.core.models.tasks.Output
size integer
The number of logs fetched.
