Script icon
Docker icon
If icon
SlackIncomingWebhook icon
Log icon
Return icon
Schedule icon

Tekton PipelineRun TaskRun Timeout and PVC Exhaustion Sentinel

Audits Tekton CI/CD controller Prometheus metrics to detect failed PipelineRuns, TaskRun execution timeouts, and PVC bottlenecks.

Categories
CoreInfrastructure

Automates continuous health and performance surveillance for Tekton cloud-native CI/CD Pipelines controllers.

How it works

  1. A schedule trigger executes the Tekton CI/CD pipeline audit workflow every 15 minutes.
  2. The Python task queries the Tekton controller Prometheus metrics endpoint on port 9090.
  3. The script extracts active PipelineRun executions, failed build runs, TaskRun step timeouts, and workspace duration metrics.
  4. An evaluation gate determines whether pipeline failures or task timeouts breach operational tolerance bounds.
  5. When thresholds are breached, a high-priority Slack alert is dispatched to CI/CD engineering and platform SREs.
  6. When builds run cleanly without task timeouts, nominal telemetry is logged.
  7. A structured JSON audit manifest is emitted for developer delivery SLA tracking and build governance archives.

Architecture diagram

graph TD
    Trigger[Schedule Trigger: 15m] --> ScanMetrics[Python Script: Scrape Tekton Metrics]
    ScanMetrics --> EvalBreach{Failed Runs > Threshold OR Timeouts > 2?}
    EvalBreach -- Yes --> SlackAlert[Slack: High-Urgency Build Alert]
    EvalBreach -- No --> LogNominal[Log: Nominal Pipeline State]
    SlackAlert --> EmitManifest[Debug Return: Tekton Audit Manifest]
    LogNominal --> EmitManifest

Architecture components

  • Trigger: Automated schedule polling at 15-minute intervals.
  • Script Task: Scrapes Tekton Prometheus telemetry endpoints using Docker container isolation.
  • Decision Gate: Evaluates whether build failure rates or step execution timeouts exceed delivery limits.
  • Notification Task: Dispatches structured Slack alerts with namespace identity and error counts.
  • Audit Manifest: Emits JSON execution payloads containing build throughput health and timestamps.

Key capabilities

  • Continuous surveillance of Tekton PipelineRun success ratios and execution failure surges.
  • Rapid detection of TaskRun step execution timeouts caused by workspace PVC disk exhaustion or hang loops.
  • Automated calculation of build failure rates across designated development and staging namespaces.
  • Zero external dependency execution via isolated containerized Python runners.

What you get

  • Preemptive alerting on CI/CD build bottlenecks before developer delivery cycles stall.
  • Automated tracking of workspace persistent volume claim contention and execution timeouts.
  • Real-time audit manifests capturing build pipeline health for SOC 2 software delivery compliance.
  • Configurable tolerance thresholds adaptable to high-throughput deployment environments.

Who it's for

  • CI/CD Engineers maintaining developer build pipelines and automated deployment workflows on Kubernetes.
  • Platform Engineering teams safeguarding shared cluster build infrastructure against run runaway costs.
  • Release Managers tracking continuous delivery throughput and build completion SLA metrics.
  • DevOps Engineers troubleshooting intermittent pipeline stalls and workspace volume mount bottlenecks.

Why orchestrate this with Kestra

  • Deterministic Scheduling: Guarantees regular pipeline health audits without fragile local crontab scripts.
  • Containerized Runners: Runs Python tasks inside isolated Docker containers without host library dependencies.
  • Condition Branching: Flowable If tasks route alerts and nominal logging cleanly without spaghetti code.
  • Enterprise Secret Safety: Securely injects Slack webhook credentials without plain-text exposure.

Use cases

  • TaskRun Timeout Burst Alerting: Alerts operations when prolonged dependency downloads or deadlocks cause task timeouts.
  • PVC Disk Full Detection: Identifies when workspace persistent volume claims fill up, preventing subsequent step execution.
  • Flaky Build Pipeline Auditing: Computes historical pipeline failure rates to pinpoint fragile build configurations.
  • Software Delivery SLA Reporting: Feeds daily pipeline throughput and success metrics into compliance audit archives.

Inputs

Name Type Default Description
tekton_metrics_url STRING http://tekton-pipelines-controller.tekton-pipelines.svc.cluster.local:9090 Base HTTP URL exposing Tekton Prometheus metrics.
cicd_namespace STRING production-cicd Target Kubernetes namespace hosting Tekton pipelines.
max_failed_pipelineruns_threshold INT 3 Max allowable failed PipelineRun executions before alerting.
max_timed_out_tasks_threshold INT 2 Max allowable TaskRun timeout events tolerated in a cycle.
alert_on_any_timeout BOOL false Whether any nonzero TaskRun timeout triggers an immediate alert.

Expected outputs

  • {{ outputs.scan_tekton_telemetry.vars.pipelineruns_total }}: Total PipelineRuns recorded in the audit window.
  • {{ outputs.scan_tekton_telemetry.vars.pipelineruns_failed_total }}: Count of failed PipelineRun executions.
  • {{ outputs.scan_tekton_telemetry.vars.taskruns_total }}: Total TaskRuns evaluated across pipelines.
  • {{ outputs.scan_tekton_telemetry.vars.taskruns_timed_out_total }}: Count of timed out TaskRun executions.
  • {{ outputs.scan_tekton_telemetry.vars.is_alerting }}: Boolean flag indicating reliability threshold violation.
  • {{ outputs.emit_audit_manifest.value }}: Structured JSON audit manifest emitted for compliance archives.

Prerequisites

  • Tekton Pipelines controller deployed with Prometheus metrics enabled on port 9090.
  • Metrics port reachable from the Kestra worker execution environment.
  • Docker daemon running on the Kestra host to execute containerized Python tasks.

Secrets

  • SLACK_WEBHOOK_URL: Incoming webhook URL for the target CI/CD Engineering Slack alerting channel.

Quick start

  1. Ensure Tekton Pipelines controller exposes port 9090 for Prometheus metrics scraping.
  2. Configure the SLACK_WEBHOOK_URL secret in your Kestra namespace settings.
  3. Specify your CI/CD namespace in cicd_namespace.
  4. Execute a manual test run via the Kestra UI to verify metric ingestion.
  5. Enable the periodic_tekton_audit trigger to establish automated continuous surveillance.

Common pitfalls and troubleshooting

  • Port Accessibility: Ensure Kubernetes NetworkPolicies permit TCP traffic on port 9090 from Kestra.
  • Controller Multi-Tenancy: If multiple Tekton controllers run in different namespaces, specify the exact target controller URL.
  • Metric Cardinality: Extremely large clusters with millions of historical TaskRuns may require Prometheus scraping optimizations.
  • Workspace PVC Locking: High concurrency builds sharing ReadWriteOnce PVCs can stall; inspect volume claim bindings.

How to extend

  • Automated Task Cancellation: Chain a kubectl task to cancel hung PipelineRuns exceeding maximum execution time limits.
  • Dynamic Volume Resizing: Trigger automated PVC expansion flows when build workspaces reach 90% disk utilization.
  • PagerDuty Escalation: Escalate to on-call release engineers if build failures breach 25% within an hour.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.