Log Synchronisation send logs to multiple desired destinations like ELS, datadog, Azure...
The log synchronisation task query logs and use LogShipper plugins to send them to a destination like Elasticsearch.
An execution use either the last log send date or now minus startingDayBefore as the starting point to search logs, fetch a batch of logs (1000 per default), send them via the log shippers, then store the date of the last fetched log to used as a starting point for the next execution.
type: "io.kestra.plugin.ee.core.log.LogShipper"Send logs
id: log_shipper
namespace: system
tasks:
- id: synchronize_logs
type: io.kestra.plugin.ee.core.log.LogShipper
logLevelFilter: INFO
batchSize: 2
lookbackPeriod: P1D
offsetKey: LogShipper-state
logExporters:
- type: io.kestra.plugin.ee.core.log.FileLogShipper
id: fileLogExporter
YES1000YESINFOLog level to send
The minimum log level to send
YESP1DdurationStarting duration before now
Fetch start date will be set to now subtract this duration if this task has no previous executions or no state stored
YESNamespace to search
The namespace to use to filter logs
YESLogShipper-statePrefix of the KVStore key
The prefix of the KVStore key that contain the last execution end fetched date
NO^[a-zA-Z0-9][a-zA-Z0-9_-]*1