Gate Artifactory Quotas and Retention Before Artifacts Vanish
Score an Artifactory storage report for quota and retention violations, alert Slack with the offenders, and optionally fail the run.
Audits InfluxData Telegraf agent metrics to detect buffer saturation and dropped measurements, alerting SRE teams on Slack.
Automates continuous health surveillance of InfluxData Telegraf metric collection agents across server fleets and container clusters. Interrogates internal Telegraf Prometheus metrics to detect output buffer exhaustion, discarded data points, and downstream database ingestion bottlenecks before telemetry gaps affect observability dashboards.
internal_agent_metrics_dropped, internal_agent_metrics_gathered), calculating total dropped data points.flowchart TD
A[periodic_telegraf_audit Schedule Trigger] --> B[scan_telegraf_buffer_metrics Python Script]
B --> C[evaluate_buffer_drop_threshold Flowable If]
C -- Dropped Metrics Exceeded --> D[notify_slack_telemetry_sre Slack Webhook]
C -- Buffers Operating Nominally --> E[log_nominal_telegraf_state Core Log]
D --> F[emit_audit_manifest Debug Return]
E --> F
periodic_telegraf_audit): Automated cron trigger executing regular surveillance sweeps across production collector fleets.scan_telegraf_buffer_metrics): Dockerized Python runtime querying Telegraf internal metrics with connection timeout safeguards.evaluate_buffer_drop_threshold): Conditional branching logic isolating severe downstream backpressure from normal buffer flushes.notify_slack_telemetry_sre): Formatted Slack webhook message delivering actionable telemetry and remediation runbooks.emit_audit_manifest): Standardized JSON output recording exact run parameters for compliance and reliability reporting.metric_buffer_limit settings in telegraf.conf.| Name | Type | Default | Description |
|---|---|---|---|
telegraf_prometheus_url |
STRING |
http://telegraf-fleet-gateway.internal:9273/metrics |
Base HTTP URL of the internal Prometheus metrics endpoint exposed by Telegraf agent collector instances. |
fleet_identifier |
STRING |
production-baremetal-fleet |
Identifier of the Telegraf agent collector fleet or host group under surveillance. |
max_dropped_metrics_threshold |
INT |
20 |
Maximum allowable dropped metrics count across the lookback period before triggering alerts. |
lookback_minutes |
INT |
15 |
Time range in minutes across which Telegraf agent buffer metrics are evaluated. |
alert_on_overflow |
BOOL |
true |
Toggle whether Slack alerts are dispatched upon detecting dropped metrics or saturated agent buffers. |
{{ outputs.scan_telegraf_buffer_metrics.vars.metrics_dropped_count }}: Total number of metrics discarded due to buffer capacity exhaustion.{{ outputs.scan_telegraf_buffer_metrics.vars.metrics_gathered_count }}: Cumulative count of measurements successfully collected by agent input plugins.{{ outputs.scan_telegraf_buffer_metrics.vars.metrics_written_count }}: Total number of metrics flushed to destination output sinks.{{ outputs.scan_telegraf_buffer_metrics.vars.is_alerting }}: Boolean flag indicating whether observed dropped metrics breached tolerance limits.{{ outputs.scan_telegraf_buffer_metrics.vars.audit_timestamp }}: ISO-8601 UTC timestamp recording exact time of sentinel verification.[[outputs.prometheus_client]] listen = ":9273").SLACK_WEBHOOK_URL: Secret webhook URL for dispatching incident notifications to team Slack rooms.SLACK_WEBHOOK_URL secret within the target Kestra namespace.company.sre namespace.telegraf_prometheus_url with your internal Telegraf service hostname or load balancer.periodic_telegraf_audit trigger to commence automated background monitoring.[[inputs.internal]] is declared in telegraf.conf to expose internal agent buffer statistics.metric_buffer_limit = 10000 fills in seconds during network blips; size accordingly.flush_jitter prevents simultaneous burst drops across large collector fleets.metric_buffer_limit across affected server hosts.