Script icon
Docker icon
If icon
SlackIncomingWebhook icon
Log icon
Return icon
Schedule icon

Werf Container Build Stage Cache Sync Failure Sentinel

Audits Werf container build telemetry to detect stage cache sync failures, image layer storage bloat, and remote registry disconnects.

Categories
CloudInfrastructure

Automates continuous health and caching stability surveillance for Werf GitOps container build pipelines.

How it works

  1. A schedule trigger executes the Werf build telemetry audit workflow every 15 minutes.
  2. The Python task queries the Werf metrics service endpoint on port 9090.
  3. The script extracts stage cache synchronization errors, aggregate layer cache storage volumes, and registry connectivity.
  4. An evaluation gate determines whether sync failures, cache size, or registry errors breach operational bounds.
  5. When thresholds are breached, an incident notification is posted to the CI/CD engineering Slack channel.
  6. When builds synchronize cleanly with low cache failure rates, nominal telemetry is logged.
  7. A structured JSON audit manifest is emitted for container build infrastructure governance and cache efficiency records.

Architecture diagram

graph TD
    Trigger[Schedule Trigger: 15m] --> ScanMetrics[Python Script: Scrape Werf Metrics]
    ScanMetrics --> EvalBreach{Sync Failures > Threshold OR Registry Down?}
    EvalBreach -- Yes --> SlackAlert[Slack: High-Urgency CI Alert]
    EvalBreach -- No --> LogNominal[Log: Nominal Werf State]
    SlackAlert --> EmitManifest[Debug Return: Build Audit Manifest]
    LogNominal --> EmitManifest

Architecture components

  • Trigger: Automated schedule polling at 15-minute intervals.
  • Script Task: Scrapes Werf GitOps Prometheus metrics endpoints using Docker container isolation.
  • Decision Gate: Evaluates whether stage cache sync failure counts or layer storage sizes exceed limits.
  • Notification Task: Dispatches structured Slack alerts with pipeline namespace and cache storage metrics.
  • Audit Manifest: Emits JSON execution payloads containing container build telemetry and audit timestamps.

Key capabilities

  • Continuous surveillance of intermediate container stage caching across multi-stage Werf pipelines.
  • Rapid detection of layer push and pull synchronization failures caused by registry timeouts or rate limits.
  • Automated tracking of aggregate stage cache disk utilization to prevent storage volume exhaustion.
  • Containerized execution via official Kestra Docker task runners guaranteeing reliable environment execution.

What you get

  • Real-time visibility into GitOps container build acceleration and layer reuse efficiency.
  • Proactive alerts preventing build queue stalls caused by remote registry synchronization failures.
  • Standardized audit manifests containing exact cache sizes and pipeline execution metrics.
  • Zero-dependency deployment compatible with self-hosted Kubernetes runners and cloud agents.

Who it's for

  • Platform SREs and DevOps engineers maintaining enterprise GitOps CI/CD build infrastructure.
  • Build engineering teams optimizing container compile times and intermediate Docker layer caching.
  • Kubernetes platform architects standardizing consistent image builds across microservice repositories.
  • Release managers ensuring container artifacts publish reliably to production registries.

Why orchestrate this with Kestra

  • Serverless Docker Isolation: Executes telemetry scraping in ephemeral Docker environments without persistent workers.
  • Centralized Secrets Governance: Protects sensitive Slack webhooks using encrypted namespace secrets.
  • Declarative YAML Workflows: Manages container build cache monitoring as versioned code in Git repositories.
  • Comprehensive Audit History: Preserves historical build telemetry records in searchable Kestra execution logs.

Use cases

  • Build Acceleration Protection: Detects when cache layer misses trigger cold Docker builds across development teams.
  • Storage Quota Enforcement: Alerts platform engineers before stage cache volumes exceed cloud block storage limits.
  • Registry Latency Detection: Identifies slow network hops between CI runner clusters and artifact registries.

Inputs

Name Type Default Description
werf_metrics_url STRING http://werf-metrics.ci.svc.cluster.local:9090 Internal HTTP URL exposing Prometheus metrics for Werf.
pipeline_namespace STRING cloud-native-microservices CI/CD pipeline namespace whose stage cache is monitored.
max_cache_sync_failures_threshold INT 5 Maximum allowable stage cache sync failures within the window.
max_stage_cache_size_gb INT 150 Maximum acceptable aggregate stage cache storage in gigabytes.
alert_on_registry_unreachable BOOL true Toggle whether registry reachability failure triggers an alert.

Expected outputs

  • {{ outputs.scan_werf_cache_telemetry.vars.cache_sync_failures_count }}: Count of stage cache layer sync failures.
  • {{ outputs.scan_werf_cache_telemetry.vars.stage_cache_size_gb }}: Aggregate stage cache storage size in gigabytes.
  • {{ outputs.scan_werf_cache_telemetry.vars.registry_reachable_flag }}: Health status indicator of remote cache registry.
  • {{ outputs.scan_werf_cache_telemetry.vars.is_alerting }}: Boolean flag indicating if build cache SLA is breached.
  • {{ outputs.scan_werf_cache_telemetry.vars.audit_timestamp }}: ISO-8601 UTC timestamp of the build audit run.

Prerequisites

  • Werf 1.2+ deployed in CI/CD pipelines with Prometheus metrics exporter enabled.
  • Network connectivity from the Kestra worker to the Werf metrics service port (default 9090).
  • Configured Slack incoming webhook URL with permissions to post in CI/CD alert channels.

Secrets

  • SLACK_WEBHOOK_URL: Secure webhook URL for routing build cache alerts to the CI/CD Slack channel.

Quick start

  1. Verify that Werf metrics exporter is active on HTTP port 9090.
  2. Add the SLACK_WEBHOOK_URL secret within your Kestra namespace settings.
  3. Specify your application build namespace in pipeline_namespace.
  4. Run a manual flow execution via the Kestra UI to validate metric ingestion.
  5. Enable the periodic_werf_audit schedule trigger to establish automated monitoring.

Common pitfalls and troubleshooting

  • Registry Authentication: Sync failures often trace to expired robot account tokens for container registries.
  • Garbage Collection: If cache size grows unbounded, configure automated werf cleanup cron jobs.
  • Divergent Base Images: Excessive layer cache growth can result from frequently changing base image digests.
  • Network Throttling: Heavy parallel builds may saturate outbound NAT gateway bandwidth during layer uploads.

How to extend

  • Automated Cache Cleanup: Trigger a werf cleanup job via Kestra when the cache footprint exceeds 200 GB.
  • PagerDuty Escalation: Escalate to on-call CI engineers if remote registry disconnects persist for 3 runs.
  • Multi-Region Sync: Monitor secondary cache mirroring latencies across multi-region artifact registries.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.