GCP Incident War Room with Slack, GitHub and Telegram in Kestra

id: gcp-incident-war-room
namespace: company.team
description: |
  Turn every Google Cloud Monitoring incident into a Slack war room for the
  team that owns the alert, with an optional GitHub tracking issue and
  Telegram page, and archive it once GCP reports the incident as resolved.

variables:
  pubsub_topic: incident-alerts
  pubsub_subscription: incident-alerts-kestra
  slack_oncall_by_team:
    default: [ "U0123ABCDEF" ]
    payments: [ "U0123ABCDEF", "U0456GHIJKL" ]
  oncall_label: team
  enable_github: true
  github_repository: your-org/your-repo
  enable_telegram: true
  escalate_severities: [ "critical", "error" ]
  archive_after: PT24H

triggers:
  - id: gcp_alert
    type: io.kestra.plugin.gcp.pubsub.RealtimeTrigger
    description: One execution per Cloud Monitoring notification, both when an
      incident opens and when it closes.
    projectId: "{{ secret('GCP_PROJECT_ID') }}"
    serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT') }}"
    topic: "{{ vars.pubsub_topic }}"
    subscription: "{{ vars.pubsub_subscription }}"
    autoCreateSubscription: false
    serdeType: JSON

tasks:
  - id: route_by_state
    type: io.kestra.plugin.core.flow.Switch
    value: "{{ trigger.data.incident.state }}"
    cases:
      open:
        - id: lookup_open_incident
          type: io.kestra.plugin.core.kv.Get
          description: A re-delivered Pub/Sub message must never open a second room for
            the same incident.
          key: "incident_{{ trigger.data.incident.incident_id | slugify }}"
          errorOnMissing: false

        - id: open_once
          type: io.kestra.plugin.core.flow.If
          condition: "{{ (outputs.lookup_open_incident.value ?? '') == '' }}"
          then:
            - id: route_oncall
              type: io.kestra.plugin.core.output.OutputValues
              description: Read the team from the alerting policy's user label; use `default`
                when the label is missing or the team has no entry in
                `slack_oncall_by_team`.
              values:
                team: >-
                  {%- set label =
                  trigger.data.incident.policy_user_labels[vars.oncall_label] ??
                  'default' -%} {{- (vars.slack_oncall_by_team | keys) contains
                  label ? label : 'default' -}}

            - id: create_war_room
              type: io.kestra.plugin.slack.app.conversations.Create
              description: One channel per incident. The policy part of the name is capped at
                50 characters to stay under Slack's 80-character limit.
              token: "{{ secret('SLACK_TOKEN') }}"
              name: >-
                {%- set policy = trigger.data.incident.policy_name | slugify -%}
                inc-{{ policy | slice(0, min(50, policy | length)) }}-{{
                trigger.data.incident.incident_id | substringAfterLast('.') |
                lower }}

            - id: remember_war_room
              type: io.kestra.plugin.core.kv.Set
              description: Saved straight away so the close notification can still archive the
                channel if a later step fails.
              key: "incident_{{ trigger.data.incident.incident_id | slugify }}"
              kvType: JSON
              value: '{"channel_id": "{{ outputs.create_war_room.id }}"}'

            - id: invite_oncall
              type: io.kestra.plugin.slack.app.conversations.Invite
              token: "{{ secret('SLACK_TOKEN') }}"
              channel: "{{ outputs.create_war_room.id }}"
              users: "{{ vars.slack_oncall_by_team[outputs.route_oncall.values.team] | toJson
                }}"

            - id: post_briefing
              type: io.kestra.plugin.slack.app.chats.Post
              description: The runbook comes from the alerting policy's Documentation field.
              token: "{{ secret('SLACK_TOKEN') }}"
              channel: "{{ outputs.create_war_room.id }}"
              messageText: |
                :rotating_light: *{{ trigger.data.incident.policy_name }}*, {{ trigger.data.incident.severity ?? 'no severity' }}
                {{ trigger.data.incident.summary }}

                *Condition:* {{ trigger.data.incident.condition_name }}
                *Resource:* {{ trigger.data.incident.resource_display_name ?? trigger.data.incident.resource_name ?? 'n/a' }}
                *Observed:* {{ trigger.data.incident.observed_value ?? 'n/a' }} (threshold {{ trigger.data.incident.threshold_value ?? 'n/a' }})
                *Started:* <!date^{{ trigger.data.incident.started_at }}^{date_short_pretty} {time}|{{ trigger.data.incident.started_at }}>
                *Paged:* {% for user in vars.slack_oncall_by_team[outputs.route_oncall.values.team] %}<@{{ user }}> {% endfor %}(team: `{{ outputs.route_oncall.values.team }}`)

                *Runbook*
                {{ trigger.data.incident.documentation.content ?? 'No runbook attached to this alerting policy. Add one in the policy documentation field.' }}

                <{{ trigger.data.incident.url }}|Open the incident in Cloud Monitoring>

            - id: track_and_escalate
              type: io.kestra.plugin.core.flow.Parallel
              tasks:
                - id: track_if_enabled
                  type: io.kestra.plugin.core.flow.If
                  condition: "{{ vars.enable_github }}"
                  then:
                    - id: open_tracking_issue
                      type: io.kestra.plugin.github.issues.Create
                      oauthToken: "{{ secret('GITHUB_TOKEN') }}"
                      repository: "{{ vars.github_repository }}"
                      title: "[Incident] {{ trigger.data.incident.policy_name }}: {{
                        trigger.data.incident.condition_name }}"
                      body: |
                        **Severity:** {{ trigger.data.incident.severity ?? 'no severity' }}
                        **Summary:** {{ trigger.data.incident.summary }}
                        **Resource:** {{ trigger.data.incident.resource_display_name ?? trigger.data.incident.resource_name ?? 'n/a' }}
                        **Owning team:** {{ outputs.route_oncall.values.team }}
                        **Slack war room:** #{{ outputs.create_war_room.name }}
                        **Cloud Monitoring:** {{ trigger.data.incident.url }}

                        ## Timeline
                        - Opened: {{ (trigger.data.incident.started_at * 1000) | date("yyyy-MM-dd HH:mm 'UTC'", timeZone='UTC') }}

                        _Opened by Kestra execution `{{ execution.id }}`._

                    - id: remember_tracking_issue
                      type: io.kestra.plugin.core.kv.Set
                      key: "incident_{{ trigger.data.incident.incident_id | slugify }}"
                      kvType: JSON
                      value: '{"channel_id": "{{ outputs.create_war_room.id }}", "issue_number": {{
                        outputs.open_tracking_issue.issueNumber }}}'

                    - id: link_tracking_issue
                      type: io.kestra.plugin.slack.app.chats.Post
                      token: "{{ secret('SLACK_TOKEN') }}"
                      channel: "{{ outputs.create_war_room.id }}"
                      messageText: ":memo: Tracking issue: <{{ outputs.open_tracking_issue.issueUrl
                        }}|#{{ outputs.open_tracking_issue.issueNumber }}>"

                - id: escalate_if_severe
                  type: io.kestra.plugin.core.flow.If
                  description: Severities are compared in lowercase because Cloud Monitoring sends
                    both `Critical` and `no severity`.
                  condition: "{{ vars.enable_telegram and (vars.escalate_severities contains
                    ((trigger.data.incident.severity ?? 'no severity') | lower))
                    }}"
                  then:
                    - id: page_oncall
                      type: io.kestra.plugin.telegram.TelegramSend
                      token: "{{ secret('TELEGRAM_TOKEN') }}"
                      channel: "{{ secret('TELEGRAM_CHAT_ID') }}"
                      payload: |
                        🚨 {{ trigger.data.incident.severity }} incident: {{ trigger.data.incident.policy_name }}
                        {{ trigger.data.incident.summary }}

                        Team: {{ outputs.route_oncall.values.team }}
                        War room: #{{ outputs.create_war_room.name }}
                        {{ trigger.data.incident.url }}

          else:
            - id: skip_duplicate_open
              type: io.kestra.plugin.core.log.Log
              level: WARN
              message: "A war room already exists for incident {{
                trigger.data.incident.incident_id }}; ignoring the repeated open
                notification."

      closed:
        - id: lookup_incident
          type: io.kestra.plugin.core.kv.Get
          key: "incident_{{ trigger.data.incident.incident_id | slugify }}"
          errorOnMissing: false

        - id: close_if_known
          type: io.kestra.plugin.core.flow.If
          condition: "{{ (outputs.lookup_incident.value ?? '') != '' }}"
          then:
            - id: post_resolution
              type: io.kestra.plugin.slack.app.chats.Post
              description: Without a tracking issue the stored record has no `issue_number`,
                so it's read with `?? 0`.
              token: "{{ secret('SLACK_TOKEN') }}"
              channel: "{{ outputs.lookup_incident.value.channel_id }}"
              messageText: |
                :white_check_mark: *Resolved* after {{ (trigger.data.incident.ended_at - trigger.data.incident.started_at) / 60 }} min.
                {% if (outputs.lookup_incident.value.issue_number ?? 0) > 0 %}Tracking issue #{{ outputs.lookup_incident.value.issue_number }} is closed. {% endif %}This channel stays open for follow-up and is archived after the grace period (`{{ vars.archive_after }}`); resume the paused execution in Kestra to archive it sooner.

            - id: resolve_if_tracked
              type: io.kestra.plugin.core.flow.If
              condition: "{{ vars.enable_github and
                (outputs.lookup_incident.value.issue_number ?? 0) > 0 }}"
              then:
                - id: resolve_tracking_issue
                  type: io.kestra.plugin.core.flow.Parallel
                  tasks:
                    - id: comment_resolution
                      type: io.kestra.plugin.github.issues.Comment
                      oauthToken: "{{ secret('GITHUB_TOKEN') }}"
                      repository: "{{ vars.github_repository }}"
                      issueNumber: "{{ outputs.lookup_incident.value.issue_number }}"
                      body: "Resolved by Cloud Monitoring at {{ (trigger.data.incident.ended_at *
                        1000) | date(\"yyyy-MM-dd HH:mm 'UTC'\", timeZone='UTC')
                        }}, {{ (trigger.data.incident.ended_at -
                        trigger.data.incident.started_at) / 60 }} min after it
                        opened."

                    - id: close_issue
                      type: io.kestra.plugin.core.http.Request
                      method: PATCH
                      uri: "https://api.github.com/repos/{{ vars.github_repository }}/issues/{{
                        outputs.lookup_incident.value.issue_number }}"
                      headers:
                        Authorization: "Bearer {{ secret('GITHUB_TOKEN') }}"
                        Accept: application/vnd.github+json
                      contentType: application/json
                      body: '{"state": "closed", "state_reason": "completed"}'

            - id: forget_incident
              type: io.kestra.plugin.core.kv.Delete
              key: "incident_{{ trigger.data.incident.incident_id | slugify }}"

            - id: grace_period
              type: io.kestra.plugin.core.flow.Pause
              description: Keeps the channel open for follow-up discussion; resume it from the
                UI to archive sooner.
              pauseDuration: "{{ vars.archive_after }}"
              behavior: RESUME

            - id: archive_war_room
              type: io.kestra.plugin.slack.app.conversations.Archive
              token: "{{ secret('SLACK_TOKEN') }}"
              channel: "{{ outputs.lookup_incident.value.channel_id }}"
          else:
            - id: skip_unknown_incident
              type: io.kestra.plugin.core.log.Log
              level: WARN
              message: "No war room recorded for incident {{ trigger.data.incident.incident_id
                }}; it was probably opened before this flow was deployed."

errors:
  - id: alert_flow_failure
    type: io.kestra.plugin.core.flow.If
    description: Telegram doesn't depend on Slack being reachable; without it, DM
      the first `default` on-call engineer on Slack.
    condition: "{{ vars.enable_telegram }}"
    then:
      - id: alert_failure_telegram
        type: io.kestra.plugin.telegram.TelegramSend
        token: "{{ secret('TELEGRAM_TOKEN') }}"
        channel: "{{ secret('TELEGRAM_CHAT_ID') }}"
        payload: |
          ⚠️ The incident war room flow failed while handling a {{ trigger.data.incident.state ?? 'unknown' }} notification for "{{ trigger.data.incident.policy_name ?? 'unknown policy' }}".
          Handle the incident manually and check Kestra execution {{ execution.id }}.
          {{ trigger.data.incident.url ?? '' }}
    else:
      - id: alert_failure_slack
        type: io.kestra.plugin.slack.app.chats.Post
        token: "{{ secret('SLACK_TOKEN') }}"
        channel: "{{ vars.slack_oncall_by_team['default'][0] }}"
        messageText: |
          :warning: The incident war room flow failed while handling a {{ trigger.data.incident.state ?? 'unknown' }} notification for *{{ trigger.data.incident.policy_name ?? 'unknown policy' }}*.
          Handle the incident manually and check Kestra execution `{{ execution.id }}`.
          {{ trigger.data.incident.url ?? '' }}

outputs:
  - id: incident_id
    type: STRING
    value: "{{ trigger.data.incident.incident_id }}"
  - id: incident_state
    type: STRING
    value: "{{ trigger.data.incident.state }}"
  - id: slack_channel_id
    type: STRING
    value: "{{ outputs.create_war_room.id ??
      outputs.lookup_incident.value.channel_id ?? '' }}"
  - id: github_issue_number
    type: INT
    value: "{{ outputs.open_tracking_issue.issueNumber ??
      outputs.lookup_incident.value.issue_number ?? 0 }}"

When a Cloud Monitoring alert fires, the first minutes usually go to busywork: creating a channel, finding who is on call, digging out the runbook and opening a ticket, while production is still down. This blueprint does all of that the moment GCP opens an incident, pulls in the team that owns the alert, and cleans up after itself when GCP resolves it.

How it works

Alerting policies send notifications to a Pub/Sub notification channel. The gcp_alert realtime trigger starts one execution per message, on open and on close, and route_by_state (Switch) branches on incident.state.
On open, unless the KV store already holds the incident (so a re-delivered message never opens a second room):
- route_oncall reads the team from the policy's oncall_label user label, falling back to default.
- create_war_room and invite_oncall create #inc-<policy>-<incident id> and invite that team.
- post_briefing posts the severity, summary, observed value against the threshold, resource, who was paged, the runbook from the policy's Documentation field, and a Cloud Console link.
- track_and_escalate (Parallel) opens and links a GitHub issue (enable_github) and pages Telegram for the severities in escalate_severities (enable_telegram).
- remember_war_room saves the channel in the KV store as soon as it exists, and remember_tracking_issue adds the issue, so the close can still archive the room if a later step fails.
On close, the flow posts the time to resolve, comments on and closes the issue, waits archive_after in a Pause you can resume early from the UI, then archives the channel.
If any step fails, the errors block alerts on Telegram, or by Slack DM to the first default on-call engineer when Telegram is disabled.

What you get

A channel per incident with the owning team inside and the runbook posted. Label a policy team = payments and its incidents page the payments team.
Optional GitHub tracking and severity-aware Telegram paging. Set enable_github and enable_telegram to false for a Slack-only setup.
Clean-up and idempotency: resolved incidents close their issue and archive their channel, and duplicate notifications are ignored.
Flow outputs, so a downstream flow (for example a postmortem drafter on a Flow trigger) knows which room and issue belong to the incident:
- incident_id and incident_state: the Cloud Monitoring incident, and whether this run handled its opening or its closing.
- slack_channel_id: the war room channel.
- github_issue_number: the tracking issue, or 0 when GitHub is disabled.

Who it's for

SRE and platform teams on Google Cloud that already alert with Cloud Monitoring, especially when several teams share on-call and each alert should reach the team that owns it.

Why orchestrate this with Kestra

An incident is a lifecycle, not a single notification. Kestra reacts to Cloud Monitoring in real time through the Pub/Sub trigger, correlates the open and close events of the same incident through its KV store, holds the room open in a Pause anyone can resume from the UI, and alerts through its errors block when the automation itself breaks. A Cloud Function or a cron job can post a message, but following an incident from open to archive would also need its own state store, retries and failure alerting.

Prerequisites

PROJECT_ID=your-project
PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')
SA=kestra-incident@$PROJECT_ID.iam.gserviceaccount.com

# Topic for the alerting policies, and the subscription Kestra reads
gcloud pubsub topics create incident-alerts --project $PROJECT_ID
gcloud pubsub subscriptions create incident-alerts-kestra --topic incident-alerts --project $PROJECT_ID

# Pub/Sub notification channel (add it to your alerting policies), allowed to publish to the topic
gcloud beta monitoring channels create --type pubsub --display-name "Kestra incident war room" \
  --channel-labels topic=projects/$PROJECT_ID/topics/incident-alerts --project $PROJECT_ID
gcloud pubsub topics add-iam-policy-binding incident-alerts --project $PROJECT_ID --role roles/pubsub.publisher \
  --member serviceAccount:service-$PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com

# Service account for Kestra, limited to this one subscription; the key becomes GCP_SERVICE_ACCOUNT
gcloud iam service-accounts create kestra-incident --project $PROJECT_ID
gcloud pubsub subscriptions add-iam-policy-binding incident-alerts-kestra --project $PROJECT_ID \
  --member serviceAccount:$SA --role roles/pubsub.subscriber
gcloud iam service-accounts keys create kestra-incident.json --iam-account $SA --project $PROJECT_ID

autoCreateSubscription is off because the subscription already exists. With it on, the trigger lists every subscription in the project at startup, which needs project-wide Pub/Sub permissions this service account deliberately doesn't have.

On each alerting policy, write the runbook in Documentation and add a user label for the owning team, for example team = payments.

Slack: an app with the bot scopes channels:manage, channels:read and chat:write. Copy its xoxb-… token.
GitHub (optional): a fine-grained token limited to the tracking repository, with Issues: Read and write.
Telegram (optional): a bot from @BotFather and the on-call chat ID. Send the bot a message first: bots cannot start conversations.

Secrets

GCP_PROJECT_ID: project that hosts the topic and subscription.
GCP_SERVICE_ACCOUNT: JSON key of the kestra-incident service account.
SLACK_TOKEN: Slack bot token.
GITHUB_TOKEN: GitHub token, only with enable_github.
TELEGRAM_TOKEN and TELEGRAM_CHAT_ID: Telegram bot token and chat, only with enable_telegram.

Variables

pubsub_topic, pubsub_subscription: the topic and subscription created above.
slack_oncall_by_team: team name to Slack member IDs (Profile → ⋮ → Copy member ID). A default entry with at least one ID is required.
oncall_label: the alerting policy user label that holds the team. Defaults to team.
enable_github, github_repository: open a tracking issue per incident in owner/repo.
enable_telegram, escalate_severities: page on Telegram for these severities, written in lowercase. Defaults to critical and error.
archive_after: ISO 8601 duration a resolved room stays open. Defaults to PT24H.

Quick start

Complete the prerequisites, add the secrets to your namespace and set the variables.
Save the flow; the realtime trigger starts listening immediately.

Test without breaking anything by publishing a sample incident:

gcloud pubsub topics publish incident-alerts --project $PROJECT_ID --message '{"version":"1.2","incident":{"incident_id":"0.demo123","state":"open","started_at":1767225600,"policy_name":"Demo policy","condition_name":"Demo condition","summary":"Demo incident","severity":"Critical","url":"https://console.cloud.google.com/monitoring/alerting","documentation":{"content":"Demo runbook"}}}'

Publish it again with "state":"closed" and an ended_at timestamp to run the resolution branch, and add "policy_user_labels":{"team":"payments"} to try team routing.

How to extend

Look up the current on-call from PagerDuty or Opsgenie in route_oncall instead of the slack_oncall_by_team map.
Attach the last hour of the alerting metric to the briefing with io.kestra.plugin.gcp.monitoring.Query.
Chain a postmortem drafter on the flow outputs with an io.kestra.plugin.core.trigger.Flow trigger.

Open a Slack War Room for Every Google Cloud Monitoring Incident

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Variables

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Open a Slack War Room for Every Google Cloud Monitoring Incident

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Variables

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Multisig Bitcoin Treasury Spends with PSBT Co-Signing

AWS ECR CI/CD Build and Push Docker Images to AWS Elastic Container Registry

Cassandra Superuser Login Exposure Sentinel

Camunda Process Outcome Compliance Gate

Cassandra Keyspace Replication Audit Gate

Rebuild and Push Your Docker Image When Its Base Image Is Republished

More Cloud Blueprints

Cloudflare Canary Deployment Blueprint

AWS ECR CI/CD Build and Push Docker Images to AWS Elastic Container Registry

Docker Registry Image Tag Immutability Guard

Audit Stale EBS Snapshots for Reclaimable Waste

Google Cloud CI/CD, Build and Push Docker Images to Google Cloud Artifact Registry

Query data from Druid and load it to Redshift

New to Kestra?