Script icon
Docker icon
If icon
SlackIncomingWebhook icon
Log icon
Return icon
Schedule icon

Harbor Registry Garbage Collection Execution Stall Sentinel

Audits Harbor container registry API to detect stalled garbage collection jobs and read-only lock freezes, alerting DevOps on Slack.

Categories
CloudInfrastructure

Automates continuous health and maintenance surveillance of Harbor enterprise container registry deployments. Interrogates the Harbor REST API to detect hung garbage collection (GC) executions, lingering read-only registry locks, and storage reclamation stalls before container image build pipelines and deployment rollouts freeze.

How it works

  1. Automated Scheduled Polling: The schedule trigger initiates a surveillance cycle every 4 hours to inspect Harbor garbage collection history.
  2. API Status Query: An isolated Python task authenticates against the Harbor API, extracting recent GC execution state, duration, and read-only flags.
  3. Stall Duration Calculation: Measures running job duration against configured operational ceilings to identify hung or deadlocked processes.
  4. Safety Boundary Gate: Evaluates whether active GC execution has exceeded the duration threshold or left the registry in an orphaned read-only state.
  5. Incident Notification: Emits formatted Slack alerts to DevOps channels with exact execution identifiers, running duration, and lock status.
  6. Manifest Publication: Outputs a structured machine-readable JSON object recording registry maintenance health for DevOps operational archives.

Architecture diagram

flowchart TD
    A[periodic_harbor_audit Schedule Trigger] --> B[scan_harbor_gc_status Python Script]
    B --> C[evaluate_stall_threshold Flowable If]
    C -- GC Execution Stalled --> D[notify_slack_devops_team Slack Webhook]
    C -- GC Operating Nominally --> E[log_nominal_harbor_state Core Log]
    D --> F[emit_audit_manifest Debug Return]
    E --> F

Architecture components

  • Trigger (periodic_harbor_audit): Automated cron trigger executing regular surveillance sweeps across Harbor registry instances.
  • Inspection Task (scan_harbor_gc_status): Dockerized Python runtime querying Harbor REST APIs with connection timeout safeguards.
  • Evaluation Gate (evaluate_stall_threshold): Conditional branching logic isolating long-running GC stalls from routine pruning runs.
  • Notification Action (notify_slack_devops_team): Formatted Slack webhook message delivering actionable telemetry and remediation runbooks.
  • Manifest Return (emit_audit_manifest): Standardized JSON output recording exact run parameters for compliance and reliability reporting.

Key capabilities

  • Read-Only Lock Detection: Identifies when a failed GC job leaves the container registry locked in read-only mode, blocking developer docker push commands.
  • Storage Reclamation Auditing: Validates that untagged manifests and orphaned blob layers are purged without manual administrator intervention.
  • Zero-Touch Execution: Operates agentlessly by polling standard Harbor v2.0 REST APIs over secure internal HTTPS networks.
  • CI/CD Pipeline Safeguarding: Prevents build pipeline failures caused by unexpected registry maintenance window overruns.

What you get

  • Pre-built sentinel protecting container image registries from silent garbage collection deadlocks and read-only lock freezes
  • Isolated Docker execution environment with explicit network timeout safeguards preventing stalled workflow runs
  • Real-time Slack notifications detailing exact API endpoints, execution IDs, and duration metrics
  • Standardized audit trail supporting post-incident reviews and registry storage capacity planning

Who it's for

  • DevOps Engineers maintaining enterprise Harbor container registries and Helm chart repositories
  • Platform Engineers responsible for CI/CD container image build infrastructure and developer velocity
  • Cloud Infrastructure Architects managing high-availability object storage backends for container registries

Why orchestrate this with Kestra

  • Centralizes registry maintenance surveillance without maintaining custom alerting scripts or external daemon checkers
  • Encrypts sensitive notification webhooks and registry administrator credentials using enterprise namespace secrets
  • Provides historical execution archives to satisfy compliance auditors requiring proof of continuous infrastructure health
  • Enables automated multi-system remediation such as resetting read-only locks or cancelling stalled GC executions

Use cases

  • Orphaned Read-Only Registry Lock: Detect when a Harbor worker pod crashes during GC execution, leaving the registry permanently unable to accept image pushes.
  • High-Blob-Volume GC Stalls: Identify when massive repositories with millions of untagged image layers cause GC jobs to exceed allocated maintenance windows.
  • Storage Driver Timeout Hangs: Pinpoint object storage driver deadlocks (S3, Swift, Azure Blob) where blob deletion requests hang indefinitely.

Inputs

Name Type Default Description
harbor_api_url STRING https://harbor.internal.enterprise.com/api/v2.0 Base HTTPS URL of the Harbor container registry REST API endpoint.
max_execution_hours INT 4 Maximum allowable duration in hours a garbage collection execution may remain running before alerting.
alert_on_stall BOOL true Toggle whether Slack alerts are dispatched upon detecting stalled garbage collection executions.

Expected outputs

  • {{ outputs.scan_harbor_gc_status.vars.latest_execution_id }}: Integer identifier of the most recent Harbor garbage collection execution job.
  • {{ outputs.scan_harbor_gc_status.vars.gc_job_status }}: Execution state string returned by Harbor (RUNNING, SUCCESS, ERROR, or STOPPED).
  • {{ outputs.scan_harbor_gc_status.vars.execution_duration_hours }}: Calculated elapsed time in hours of the active garbage collection execution.
  • {{ outputs.scan_harbor_gc_status.vars.is_read_only_locked }}: Boolean flag indicating whether the registry currently enforces a read-only lock.
  • {{ outputs.scan_harbor_gc_status.vars.is_alerting }}: Boolean flag indicating whether observed execution metrics breached tolerance limits.

Prerequisites

  • Harbor container registry deployed with the REST API accessible over HTTPS.
  • An authorized Harbor user account with administrative permissions to read /system/gc endpoints.
  • Incoming Slack Webhook configured with publishing permissions into the designated DevOps alerting channel.

Secrets

  • HARBOR_ADMIN_USERNAME: Secret username for authenticating to the Harbor REST API.
  • HARBOR_ADMIN_PASSWORD: Secret password or token for authenticating to the Harbor REST API.
  • SLACK_WEBHOOK_URL: Secret webhook URL for dispatching incident notifications to team Slack rooms.

Quick start

  1. Register the HARBOR_ADMIN_USERNAME, HARBOR_ADMIN_PASSWORD, and SLACK_WEBHOOK_URL secrets in Kestra.
  2. Import this blueprint into Kestra under the company.devops namespace.
  3. Update harbor_api_url with your internal Harbor service URL.
  4. Execute a manual run to verify API authentication and webhook formatting.
  5. Enable the periodic_harbor_audit trigger to commence automated background monitoring.

Common pitfalls and troubleshooting

  • Self-Signed TLS Certificates: If Harbor uses internal CA certificates, ensure Python requests accepts the custom CA bundle or set verify=False for internal endpoints.
  • Read-Only Mode Semantics: Harbor's read-only mode rejects all manifest pushes; verify whether read_only was manually toggled by an administrator.
  • API Version Differences: Ensure your Harbor instance runs v2.0 or later; older v1.x instances expose different GC endpoint routes.
  • Job Service Worker Deadlocks: If Harbor's jobservice container runs out of worker threads, GC executions will queue indefinitely without starting.

How to extend

  • Automated GC Job Cancellation: Trigger a sub-flow to execute PUT /system/gc/{id} to abort hung executions automatically.
  • Read-Only Lock Reset: Chain an administrative task to clear the read-only schedule parameter when a stalled job is cleared.
  • PagerDuty Escalation: Escalate to on-call platform engineers via PagerDuty if read-only lock persists during core business hours.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.