Script icon
Process icon
If icon
SlackIncomingWebhook icon
Log icon
Return icon
Schedule icon

Codecov Coverage Drop Regression Guard

Audits Codecov repository test coverage metrics to detect negative regressions and enforce engineering test suite coverage baselines.

Categories
CoreInfrastructure

Automated unit and integration test suites are the first line of defense against production regression bugs and system downtime. In fast-paced engineering organizations, features merged without accompanying tests gradually erode overall test coverage, increasing the likelihood of undetected edge-case bugs. This sentinel continuously interrogates the Codecov API v2, compares latest commit test coverage against previous baseline commits, flags negative coverage deltas or drops below absolute thresholds, and alerts Engineering Teams on Slack before untested code reaches production.

How it works

  1. Scheduled triggering: Executes daily at 08:00 UTC using a cron schedule trigger (0 8 * * *) or triggers via CI/CD post-merge webhooks.
  2. Commit coverage enumeration: Queries the Codecov API v2 (/api/v2/github/{owner}/repos/{repo}/commits) using a personal or organizational API token.
  3. Delta regression calculation: Compares the latest commit coverage percentage against the previous commit to compute the exact coverage delta.
  4. Condition evaluation: An io.kestra.plugin.core.flow.If task evaluates whether the coverage delta exceeds max_tolerated_coverage_drop_percentage or falls below min_absolute_coverage_percentage.
  5. Slack alert dispatch: Sends a high-priority incident payload via SlackIncomingWebhook listing commit SHAs, current coverage, and delta drop percentages.
  6. Audit trail logging and return: Emits operational execution logs and returns a structured compliance manifest for auditing.

Architecture diagram

flowchart TD
  A[scheduled_coverage_audit Schedule Trigger] --> B[scan_codecov_metrics Python Script]
  B --> C[check_coverage_regression Flowable If]
  C -- Coverage Regressed --> D[notify_slack_qa_engineering Webhook]
  C -- Coverage Preserved --> E[log_codecov_audit Log]
  D --> E
  E --> F[export_codecov_manifest Return]

Use cases

  • Continuous testing governance: Prevent gradual degradation of test suite coverage across high-velocity microservice repositories.
  • Release readiness validation: Confirm that critical release candidate branches meet minimum testing coverage thresholds before production deployments.
  • Engineering KPI tracking: Maintain historical test coverage metrics across repositories to measure team testing discipline and quality adherence.

Inputs

Name Type Default Description
repo_owner STRING enterprise-org GitHub organization or username owning the repository.
repo_name STRING core-platform-service Target repository name in Codecov to audit for code coverage regressions.
max_tolerated_coverage_drop_percentage FLOAT 1.0 Maximum allowable negative percentage drop in test coverage between commit builds before alerting.
min_absolute_coverage_percentage FLOAT 80.0 Minimum required total test coverage percentage required for repository health.
alert_on_coverage_regression BOOL true Whether to dispatch alerts when test coverage drops or falls below absolute baselines.

Expected outputs

  • {{ outputs.scan_codecov_metrics.vars.repository }}: Full repository path evaluated during the audit.
  • {{ outputs.scan_codecov_metrics.vars.commit_sha }}: Short SHA of the most recent evaluated commit.
  • {{ outputs.scan_codecov_metrics.vars.current_coverage_percentage }}: Current test coverage percentage on the audited commit.
  • {{ outputs.scan_codecov_metrics.vars.coverage_delta_percentage }}: Percentage change in coverage relative to the parent commit.
  • {{ outputs.export_codecov_manifest.value }}: Structured dictionary containing the complete test coverage audit report.

Prerequisites

  • Outbound HTTPS network access from Kestra execution workers to api.codecov.io.
  • Codecov API token with read permissions to repository metrics.
  • Incoming Slack Webhook configured for QA and Engineering alerting.

Secrets

  • CODECOV_API_TOKEN: API token authorized to read repository coverage metrics from Codecov.
  • SLACK_WEBHOOK_URL: Slack Incoming Webhook endpoint URL used for sending operational alerts.

Quick start

  1. Define CODECOV_API_TOKEN and SLACK_WEBHOOK_URL in your Kestra Secret storage.
  2. Deploy this flow under the company.devops namespace.
  3. Configure repo_owner and repo_name to target your production repository.
  4. Run a manual execution to inspect commit coverage trends and verify Slack alerting.

Common pitfalls and troubleshooting

  • Codecov upload processing lag: Codecov parses coverage reports asynchronously; allow 1 to 2 minutes after CI completes before interrogating the API.
  • Branch specification: Ensure the API request queries commits on the default or production branch (?branch=main) rather than transient feature branches.
  • Partial test runs: CI workflows with skipped test suites or partial matrix executions may falsely report sudden coverage drops; ensure full test runs are completed before metrics are fetched.

How to extend

  • Automated PR comment: Post an automated reminder on GitHub pull requests when merged code results in a detected coverage regression.
  • Deployment pipeline block: Chain this flow as a prerequisite check before triggering production deployment workflows in Kestra.
  • Multi-repository fleet scanning: Iterate across all microservice repositories in an engineering organization using a Kestra ForEach task.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.