Buf Protobuf Breaking-Change Deployment Gate for Kestra

id: buf-protobuf-breaking-change-deployment-gate
namespace: company.data

description: |
  Compare a new .proto schema against its previous merged version with
  buf's own breaking-change detector, separate changes that break wire
  compatibility or remove something a client still depends on from
  changes that only affect reflection/JSON serialization, and fail the
  execution outright on the first group so a chained CI/CD pipeline is
  actually blocked, while the second group only pages Slack for review.
  A schema that fails to even compile is treated as its own distinct
  failure mode, not silently reported as zero breaking changes.

inputs:
  - id: previous_proto_file
    type: FILE
    displayName: "Previous .proto File"
    description: "The last merged version of the schema, used as the compatibility
      baseline."
  - id: current_proto_file
    type: FILE
    displayName: "Current .proto File"
    description: "The new, pending version of the schema. Used when this flow is run
      manually or chained as a Subflow; ignored when the webhook trigger fires,
      which supplies the new schema instead."

triggers:
  - id: check_on_pr_webhook
    type: io.kestra.plugin.core.trigger.Webhook
    description: "Lets a CI pipeline POST the new .proto file directly to this flow
      as part of a pull request check. fetchType: STORE streams the POST body
      into Kestra's internal storage and exposes it as the safe trigger.uri
      reference, rather than as raw trigger.body text, which the Webhook
      trigger's own documentation warns is untrusted input unsafe to interpolate
      into a script task's commands."
    key: "{{ secret('SCHEMA_CHECK_WEBHOOK_KEY') }}"
    fetchType: STORE
    disabled: true

tasks:
  - id: check_breaking_changes
    type: io.kestra.plugin.scripts.shell.Commands
    description: |
      Runs buf's own breaking-change detector between the previous and current schema inside its container.
      buf CLI v2 requires a minimal buf.yaml to treat a bare directory as a module, so one is generated for
      both the current workspace and the baseline directory. buf breaking's --error-format=json output is
      newline-delimited JSON (one object per line), not a single array, so it is converted with awk before
      Kestra's jq filter (which expects one JSON document) ever reads it. stdout (violations) and stderr
      (a schema compile error, which is a completely different failure mode from a breaking change) are
      captured to separate files so classification can tell them apart; exit code alone cannot, since buf
      breaking returns the same exit code 1 for both a validation failure and a tool/compile error.
    containerImage: bufbuild/buf:latest
    commands:
      - "mkdir -p baseline"
      - "printf 'version: v2\\nmodules:\\n  - path: .\\n' > buf.yaml"
      - "printf 'version: v2\\nmodules:\\n  - path: .\\n' > baseline/buf.yaml"
      - "cp {{ trigger.uri is defined ? trigger.uri : inputs.current_proto_file
        }} schema.proto"
      - "cp {{ inputs.previous_proto_file }} baseline/schema.proto"
      - "buf breaking --against baseline --error-format=json > ndjson.txt 2>
        stderr.txt || true"
      - "awk 'BEGIN{print \"[\"} NR>1{print \",\"} {print} END{print \"]\"}'
        ndjson.txt > report.json"
    outputFiles:
      - "report.json"
      - "stderr.txt"
    allowFailure: true

  - id: classify_schema_compatibility
    type: io.kestra.plugin.core.debug.Return
    description: "Classify UNREACHABLE when the container itself could not produce
      output files at all, BLOCK when the schema failed to compile (non-empty
      stderr.txt) or at least one finding removes something still in use (every
      *_NO_DELETE rule) or breaks wire-level binary compatibility
      (FIELD_SAME_TYPE, RPC_SAME_REQUEST_TYPE, RPC_SAME_RESPONSE_TYPE), WARN
      when only reflection/JSON-serialization findings exist, otherwise PASS."
    format: "{{ (outputs.check_breaking_changes.outputFiles['report.json'] is
      defined and outputs.check_breaking_changes.outputFiles['stderr.txt'] is
      defined) ?
      (((read(outputs.check_breaking_changes.outputFiles['stderr.txt']) |
      length) > 0 or
      ((read(outputs.check_breaking_changes.outputFiles['report.json']) |
      jq('[.[] | select(.type | test(\"_NO_DELETE$\") or . ==
      \"FIELD_SAME_TYPE\" or . == \"RPC_SAME_REQUEST_TYPE\" or . ==
      \"RPC_SAME_RESPONSE_TYPE\")] | length') | first | default(0)) > 0)) ?
      'BLOCK' :
      (((read(outputs.check_breaking_changes.outputFiles['report.json']) |
      jq('length') | first | default(0)) > 0) ? 'WARN' : 'PASS')) :
      'UNREACHABLE' }}"

  - id: route_by_schema_compatibility
    type: io.kestra.plugin.core.flow.Switch
    description: "PASS logs a quiet confirmation. WARN pages Slack with the
      non-breaking finding types but lets the execution succeed, since these
      only affect reflection or JSON serialization. BLOCK pages Slack with
      either the compile error or the specific breaking rule types and then runs
      fail_gate to deliberately fail the execution, so a parent pipeline
      chaining this flow as a Subflow schema gate is actually stopped.
      UNREACHABLE pages Slack separately."
    value: "{{ outputs.classify_schema_compatibility.value }}"
    cases:
      "PASS":
        - id: log_pass
          type: io.kestra.plugin.core.log.Log
          message: "No breaking or wire-incompatible changes found between the previous
            and current .proto schema."
      "WARN":
        - id: alert_warn
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {
              "text": "Buf schema gate WARN: non-breaking findings only (reflection/JSON serialization, not wire compatibility): {{ read(outputs.check_breaking_changes.outputFiles['report.json']) | jq('[.[] | .type] | unique | join(", ")') | first }}. Execution {{ execution.id }}."
            }
      "BLOCK":
        - id: alert_block
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {
              "text": "Buf schema gate BLOCK: {{ (read(outputs.check_breaking_changes.outputFiles['stderr.txt']) | length) > 0 ? ('schema failed to compile: ' ~ read(outputs.check_breaking_changes.outputFiles['stderr.txt'])) : ('breaking changes found: ' ~ (read(outputs.check_breaking_changes.outputFiles['report.json']) | jq('[.[] | select(.type | test("_NO_DELETE$") or . == "FIELD_SAME_TYPE" or . == "RPC_SAME_REQUEST_TYPE" or . == "RPC_SAME_RESPONSE_TYPE") | .type] | unique | join(", ")') | first)) }}. Execution {{ execution.id }}."
            }
        - id: fail_gate
          type: io.kestra.plugin.core.execution.Fail
          description: "Deliberately fails the execution so a parent pipeline that chains
            this flow as a Subflow schema gate is actually blocked, not just
            notified."
          errorMessage: "Schema gate blocked: {{
            (read(outputs.check_breaking_changes.outputFiles['stderr.txt']) |
            length) > 0 ? 'the new .proto file failed to compile against buf' :
            ('breaking change(s) found: ' ~
            (read(outputs.check_breaking_changes.outputFiles['report.json']) |
            jq('[.[] | select(.type | test(\"_NO_DELETE$\") or . ==
            \"FIELD_SAME_TYPE\" or . == \"RPC_SAME_REQUEST_TYPE\" or . ==
            \"RPC_SAME_RESPONSE_TYPE\") | .type] | unique | join(\", \")') |
            first)) }}. Check stderr.txt or report.json in the task log before
            reapplying."
      "UNREACHABLE":
        - id: alert_unreachable
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {
              "text": "Buf schema gate UNREACHABLE: check_breaking_changes produced no output files, buf's container could not run. Execution {{ execution.id }}."
            }
    defaults:
      - id: log_unclassified
        type: io.kestra.plugin.core.log.Log
        message: "Completed buf schema compatibility check with status: {{
          outputs.classify_schema_compatibility.value }}"

  - id: log_audit_status
    type: io.kestra.plugin.core.log.Log
    description: "Post-check status line recorded whenever the execution completes
      without the BLOCK path failing it outright, for a continuous
      schema-compatibility trend in the execution history."
    message: "Buf schema gate audit: status={{
      outputs.classify_schema_compatibility.value }}, execution={{ execution.id
      }}."

errors:
  - id: alert_flow_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: "Fires both when the flow itself breaks unexpectedly and, by
      design, whenever fail_gate deliberately fails a BLOCK execution. A chained
      parent pipeline sees this flow's execution as FAILED either way."
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "text": "Buf Protobuf breaking-change deployment gate flow {{ flow.id }} FAILED (execution {{ execution.id }}). If this was a BLOCK classification, that is expected; check the execution log for the reason. If it was not, check check_breaking_changes' task log for a container problem."
      }

outputs:
  - id: status
    type: STRING
    value: "{{ outputs.classify_schema_compatibility.value }}"

A schema gate that fails on every finding buf reports trains teams to bypass it, because plenty of findings (a changed JSON field name, a changed default value) affect reflection or JSON serialization but not the binary wire format existing compiled clients and stored messages depend on. This blueprint runs buf's own breaking-change detector between a schema's previous and current version, splits findings into "breaks wire compatibility or removes something still referenced" and "affects reflection or JSON only," and only fails the execution outright on the first group, while treating a schema that fails to even compile as its own distinct failure mode rather than silently reporting zero breaking changes.

How it works

check_on_pr_webhook (io.kestra.plugin.core.trigger.Webhook) lets a CI pipeline POST a pull request's new .proto file to this flow directly. fetchType: STORE streams that POST body into Kestra's internal storage and exposes it as trigger.uri, a safe file reference, rather than as raw trigger.body text, which the trigger's own documentation flags as untrusted input unsafe to interpolate into a script task's commands. Running this flow manually or as a Subflow instead uses the current_proto_file input, the same kind of safe file reference; previous_proto_file is always supplied as an input either way.
check_breaking_changes (io.kestra.plugin.scripts.shell.Commands) runs bufbuild/buf's own container. buf CLI v2 requires a minimal buf.yaml to treat a bare directory as a module, so one is generated for both the workspace and a baseline/ directory holding the previous schema, then buf breaking --against baseline --error-format=json runs. Its JSON output is newline-delimited (one object per line), not a single array, so awk converts it to a proper array before Kestra's jq filter (which expects one JSON document) ever reads it. stdout (report.json, the violations) and stderr (stderr.txt, a schema compile error) are captured separately, because buf breaking returns the same exit code for both a validation failure and a tool/compile error, so the exit code alone cannot tell them apart.
classify_schema_compatibility (io.kestra.plugin.core.debug.Return) checks whether both output files exist; if not, UNREACHABLE. Otherwise a non-empty stderr.txt (the schema did not even compile) or any finding whose type is a *_NO_DELETE rule, FIELD_SAME_TYPE, RPC_SAME_REQUEST_TYPE, or RPC_SAME_RESPONSE_TYPE (wire-level breaking changes) classifies BLOCK; any remaining finding classifies WARN; otherwise PASS.
route_by_schema_compatibility (io.kestra.plugin.core.flow.Switch) branches on that string: PASS logs a quiet confirmation, WARN pages Slack with the non-breaking finding types but lets the execution succeed, BLOCK pages Slack with either the compile error or the specific breaking rule types and then runs fail_gate (io.kestra.plugin.core.execution.Fail) to deliberately fail the execution, and UNREACHABLE pages Slack separately.
log_audit_status records a status line whenever the execution reaches it, i.e. every outcome except BLOCK, which fails before reaching this task by design.
The errors block fires both for a genuine flow break and, intentionally, every time fail_gate fires, a chained parent pipeline sees either case as the same FAILED execution and stops.

What you get

A schema gate that distinguishes "breaks the wire format or removes something a client still calls" from "affects reflection or JSON serialization only," instead of a blanket finding-count threshold that trains people to ignore it.
A separate, correctly identified failure mode for a schema that does not even compile, instead of that case being silently misreported as zero breaking changes because buf breaking's own exit code cannot distinguish the two.
A hard stop: fail_gate makes the execution itself FAILED on a breaking change, so a parent flow using this as a Subflow task before a schema registry push or a service deploy is actually blocked, not just notified into a channel someone might not read before merging.
A status output (PASS / WARN / BLOCK / UNREACHABLE) you can also read directly if you prefer branching in the caller instead of relying on failure propagation.

Who it's for

Platform and API teams enforcing Protobuf/gRPC schema compatibility who want CI to fail loudly on an actual wire-breaking change, not on every cosmetic finding buf reports.
Teams who have tried a naive "any finding, fail" gate and abandoned it because reflection/JSON-only findings made it fail on changes the team deliberately decided to ship anyway.

Why orchestrate this with Kestra

buf produces the findings; nothing in the CLI itself enforces a policy or blocks a deploy pipeline on your terms, and its exit code conflates a real validation failure with a tool error. Kestra turns that output into an actual pass/fail decision that correctly separates those two failure modes, with a real execution-history trail of exactly which rule types triggered a block versus a warning, and Fail gives a parent pipeline a real signal to stop on instead of a Slack message it has to notice in time.

Prerequisites

Docker available to the Kestra worker, to run buf's own bufbuild/buf image, no schema registry, server, or account is required; buf compares the two schema files directly.
The previous merged .proto file and the new pending .proto file.
A Slack incoming webhook URL.

Secrets

SCHEMA_CHECK_WEBHOOK_KEY: the key segment of the webhook URL (.../executions/webhook/{namespace}/{flowId}/{key}). Only needed if you enable check_on_pr_webhook; a manual or Subflow run using current_proto_file does not need it.
SLACK_WEBHOOK_URL: Slack incoming webhook used for the WARN alert, the BLOCK alert, the unreachable alert, and the flow-failure alert.
OSS note: if your Kestra instance reads secrets from environment variables instead of a secrets backend, set SECRET_SCHEMA_CHECK_WEBHOOK_KEY and SECRET_SLACK_WEBHOOK_URL as base64, for example echo -n "<your-value>" | base64.

Quick start

Add the SLACK_WEBHOOK_URL secret, and the SCHEMA_CHECK_WEBHOOK_KEY secret if you plan to enable the webhook.
Create two .proto files: a baseline, and a modified copy that both removes an RPC method (a wire-breaking change) and renames a field's JSON name (a non-breaking change), to exercise both BLOCK and WARN in one pair (or split them into two runs).
Upload them as previous_proto_file and current_proto_file, run the flow manually, and confirm status matches what you expect.
Either enable check_on_pr_webhook and have your CI pipeline curl --data-binary @new_schema.proto the webhook URL on each pull request, or wire a Subflow task pointing at this flow into your pipeline immediately before a schema registry push or service deploy.

How to extend

Point --against at a .git#branch=main reference instead of a second file input, if the Kestra worker has access to the schema's git history, to avoid needing a separately supplied baseline file.
Add a buf lint step alongside buf breaking, so style and naming convention violations surface in the same gate without being treated as breaking changes.
Store each BLOCK's finding types in a KV store keyed by schema file name, to build a team-wide record of what got stopped and why.

Buf Protobuf Breaking-Change Deployment Gate

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Buf Protobuf Breaking-Change Deployment Gate

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Data Blueprints

Apache CouchDB Database Compaction and Conflict Resolution Gate

Audit Couchbase GSI Secondary Indexes and Detect Offline States

Couchbase TTL expiry triage sentinel

Cancel runaway Couchbase N1QL queries that starve the query service

Extract Data from Couchbase and Transform It with Pandas for Analytics

Automated LLM Web Documentation Harvester with Crawl4AI

New to Kestra?