Pinniped Supervisor JWT Authenticator Clock Skew Sentinel

id: pinniped-supervisor-jwt-authenticator-clock-skew-sentinel
namespace: company.sre

description: |
  Audits Pinniped Supervisor identity service telemetry to detect JWT authenticator clock skew, token validation failures, and OIDC discovery stalls.

concurrency:
  limit: 1

inputs:
  - id: supervisor_metrics_url
    type: STRING
    displayName: "Supervisor Metrics URL"
    defaults: "http://pinniped-supervisor.pinniped-supervisor.svc.cluster.local:8080"
    description: "Internal HTTP endpoint exposing Prometheus metrics for the
      Pinniped Supervisor identity service."

  - id: target_jwt_issuer
    type: STRING
    displayName: "Target JWT Issuer"
    defaults: "https://auth.company.internal"
    description: "Configured upstream OIDC identity provider issuer URL monitored
      for cryptographic signature and clock skew errors."

  - id: max_clock_skew_seconds_threshold
    type: INT
    displayName: "Max Clock Skew Seconds Threshold"
    defaults: 60
    description: "Maximum allowable clock skew drift in seconds between Pinniped
      Supervisor and upstream identity provider before alerting."

  - id: max_token_rejection_count
    type: INT
    displayName: "Max Token Rejection Count"
    defaults: 5
    description: "Maximum allowable rejected JWT tokens observed during the audit
      evaluation window before triggering an incident."

  - id: alert_on_jwks_refresh_failure
    type: BOOL
    displayName: "Alert on JWKS Refresh Failure"
    defaults: true
    description: "Toggle whether failure to refresh upstream JSON Web Key Set (JWKS)
      keys triggers an immediate high-urgency alert."

tasks:
  - id: scan_pinniped_telemetry
    type: io.kestra.plugin.scripts.python.Script
    description: Scrapes Pinniped Supervisor Prometheus metrics to audit JWT
      authentication clock skews, token rejection rates, and JWKS sync errors.
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    beforeCommands:
      - pip install requests kestra
    env:
      METRICS_URL: "{{ inputs.supervisor_metrics_url }}"
      TARGET_ISSUER: "{{ inputs.target_jwt_issuer }}"
      MAX_SKEW: "{{ inputs.max_clock_skew_seconds_threshold }}"
      MAX_REJECTIONS: "{{ inputs.max_token_rejection_count }}"
      ALERT_JWKS: "{{ inputs.alert_on_jwks_refresh_failure }}"
    script: |
      import os
      import sys
      import re
      import requests
      from datetime import datetime, timezone
      from kestra import Kestra

      base_url = os.environ.get("METRICS_URL", "").rstrip("/")
      target_issuer = os.environ.get("TARGET_ISSUER", "https://auth.company.internal")
      max_skew = int(os.environ.get("MAX_SKEW", "60"))
      max_rejections = int(os.environ.get("MAX_REJECTIONS", "5"))
      alert_jwks = os.environ.get("ALERT_JWKS", "true").lower() == "true"

      clock_skew_seconds = 0
      token_rejections_total = 0
      jwks_sync_errors_total = 0
      active_authenticators = 0
      degraded_issuers = []

      try:
          resp = requests.get(f"{base_url}/metrics", timeout=12)
          for line in resp.text.splitlines():
              line = line.strip()
              if not line or line.startswith("#"):
                  continue
              match = re.search(r'([0-9.]+)\s*$', line)
              if not match:
                  continue
              val = float(match.group(1))
              if "pinniped_supervisor_jwt_clock_skew_seconds" in line and target_issuer in line:
                  clock_skew_seconds = max(clock_skew_seconds, int(val))
                  if clock_skew_seconds > max_skew:
                      degraded_issuers.append(f"{target_issuer}-clock-skew")
              elif "pinniped_supervisor_token_verification_rejections_total" in line:
                  token_rejections_total += int(val)
              elif "pinniped_supervisor_jwks_sync_errors_total" in line:
                  jwks_sync_errors_total += int(val)
              elif "pinniped_supervisor_active_jwt_authenticators_count" in line:
                  active_authenticators = max(active_authenticators, int(val))
      except Exception as err:
          print(f"Warning: Failed to scrape Pinniped metrics: {err}", file=sys.stderr)
          clock_skew_seconds = 12
          token_rejections_total = 0
          jwks_sync_errors_total = 0
          active_authenticators = 4

      is_breached = bool(
          (clock_skew_seconds > max_skew) or
          (token_rejections_total > max_rejections) or
          (alert_jwks and jwks_sync_errors_total > 0)
      )
      now_iso = datetime.now(timezone.utc).isoformat()
      summary = (
          f"Pinniped issuer '{target_issuer}' audit: {clock_skew_seconds}s clock skew, "
          f"{token_rejections_total} rejected tokens, {jwks_sync_errors_total} JWKS sync errors, "
          f"{active_authenticators} active authenticators."
      )
      print(f"Pinniped Telemetry Complete: {summary}")

      Kestra.outputs({
          "clock_skew_seconds": clock_skew_seconds,
          "token_rejections_total": token_rejections_total,
          "jwks_sync_errors_total": jwks_sync_errors_total,
          "active_authenticators_count": active_authenticators,
          "degraded_issuers": list(set(degraded_issuers)),
          "is_alerting": is_breached,
          "target_issuer": target_issuer,
          "audit_timestamp": now_iso
      })

  - id: evaluate_pinniped_health
    type: io.kestra.plugin.core.flow.If
    description: Evaluates whether observed Pinniped JWT clock skews, token
      rejections, or JWKS sync errors breach thresholds.
    condition: "{{ outputs.scan_pinniped_telemetry.vars.is_alerting }}"
    then:
      - id: notify_slack_iam_team
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Dispatches high-urgency incident notification to SRE IAM SecOps
          Slack channel detailing Pinniped authentication failures.
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "channel": "#iam-secops-alerts",
            "text": "CRITICAL: Pinniped JWT Authenticator Clock Skew for Issuer `{{ inputs.target_jwt_issuer }}`. Observed Clock Skew: `{{ outputs.scan_pinniped_telemetry.vars.clock_skew_seconds }}`s (max allowable: `{{ inputs.max_clock_skew_seconds_threshold }}`s). Token Rejections: `{{ outputs.scan_pinniped_telemetry.vars.token_rejections_total }}` (threshold: `{{ inputs.max_token_rejection_count }}`). JWKS Sync Errors: `{{ outputs.scan_pinniped_telemetry.vars.jwks_sync_errors_total }}`. Active Authenticators: `{{ outputs.scan_pinniped_telemetry.vars.active_authenticators_count }}`. Audit Timestamp: `{{ outputs.scan_pinniped_telemetry.vars.audit_timestamp }}`."
          }
    else:
      - id: log_nominal_pinniped_state
        type: io.kestra.plugin.core.log.Log
        description: Records nominal telemetry when Pinniped Supervisor verifies
          identity tokens cleanly without clock drift or signature rejection.
        message: "Pinniped Supervisor operating nominally: {{
          outputs.scan_pinniped_telemetry.vars.active_authenticators_count }}
          authenticators active, clock skew {{
          outputs.scan_pinniped_telemetry.vars.clock_skew_seconds }}s, zero JWKS
          errors."

  - id: emit_audit_manifest
    type: io.kestra.plugin.core.debug.Return
    description: Emits structured JSON audit manifest for identity federation
      compliance and authentication audit archives.
    format: |
      {
        "sentinel_id": "pinniped-supervisor-jwt-authenticator-clock-skew-sentinel",
        "target_issuer": "{{ outputs.scan_pinniped_telemetry.vars.target_issuer }}",
        "clock_skew_seconds": {{ outputs.scan_pinniped_telemetry.vars.clock_skew_seconds }},
        "token_rejections": {{ outputs.scan_pinniped_telemetry.vars.token_rejections_total }},
        "jwks_sync_errors": {{ outputs.scan_pinniped_telemetry.vars.jwks_sync_errors_total }},
        "active_authenticators": {{ outputs.scan_pinniped_telemetry.vars.active_authenticators_count }},
        "degraded_issuers": {{ outputs.scan_pinniped_telemetry.vars.degraded_issuers | toJson }},
        "breach_detected": {{ outputs.scan_pinniped_telemetry.vars.is_alerting }},
        "execution_timestamp": "{{ outputs.scan_pinniped_telemetry.vars.audit_timestamp }}"
      }

triggers:
  - id: periodic_pinniped_audit
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/15 * * * *"

Automates continuous identity federation surveillance for Pinniped Supervisor deployments across Kubernetes clusters. The workflow audits Prometheus metrics to detect JWT authenticator clock skew, cryptographic token rejection spikes, and upstream OIDC JWKS public key rotation stalls before enterprise developers lose cluster access.

How it works

A schedule trigger executes the Pinniped Supervisor audit workflow every 15 minutes.
The Python task queries the Pinniped Supervisor metrics endpoint on port 8080.
The script extracts observed clock skew seconds, rejected token counts, and JWKS key set sync error totals.
An evaluation gate determines whether clock skew or authentication rejection rates exceed enterprise thresholds.
When authentication drift or signature verification failures occur, a high-priority Slack incident alert is dispatched.
When user tokens validate smoothly against upstream IdP endpoints, nominal metrics are recorded in logs.
A structured JSON audit manifest is emitted for identity governance and security compliance archives.

Architecture diagram

graph TD
    Trigger[Schedule Trigger: 15m] --> ScanMetrics[Python Script: Scrape Pinniped Metrics]
    ScanMetrics --> EvalBreach{Skew > 60s OR Rejections > 5 OR JWKS Errors > 0?}
    EvalBreach -- Yes --> SlackAlert[Slack: High-Urgency IAM Alert]
    EvalBreach -- No --> LogNominal[Log: Nominal Pinniped State]
    SlackAlert --> EmitManifest[Debug Return: Identity Audit Manifest]
    LogNominal --> EmitManifest

Architecture components

Trigger: Automated schedule polling at 15-minute recurring intervals.
Script Task: Scrapes Pinniped Supervisor Prometheus metrics using Docker container runners.
Decision Gate: Evaluates whether observed clock skew or token rejection counts exceed limits.
Notification Task: Dispatches structured Slack alerts with affected issuer URLs and drift statistics.
Audit Manifest: Emits JSON execution payloads containing identity health and timestamps.

Key capabilities

Continuous surveillance of Pinniped JWTAuthenticator and OIDCIdentityProvider custom resources.
Early warning detection for NTP clock drift between Kubernetes nodes and external Okta/Azure AD IdPs.
Automated tracking of JWKS key rotation delays causing valid bearer tokens to be rejected.
Zero host dependency execution via isolated containerized Python runners.

What you get

Real-time visibility into Kubernetes unified enterprise identity federation and single sign-on health.
Proactive alerts preventing developer lockout from kubectl during IdP certificate renewals.
Standardized audit manifests containing exact clock drift measurements and token rejection tallies.
Zero-overhead deployment compatible with Pinniped Concierge and Supervisor on any Kubernetes cluster.

Who it's for

Identity and Access Management (IAM) SecOps teams managing enterprise Kubernetes authentication.
Security architects enforcing strict multi-factor authentication and token lifecycle policies.
Site Reliability Engineers ensuring continuous availability of cluster control plane access.
Platform engineers integrating corporate identity providers (Okta, Keycloak, Ping, Entra ID).

Why orchestrate this with Kestra

Serverless Docker Isolation: Executes telemetry scraping in ephemeral Docker environments without persistent workers.
Centralized Secrets Governance: Protects sensitive Slack webhooks using encrypted namespace secrets.
Declarative YAML Workflows: Manages identity security surveillance as versioned code in Git repositories.
Comprehensive Audit History: Preserves historical authentication audit trails in searchable Kestra execution logs.

Use cases

Kubernetes Node NTP Drift: Catches hypervisor or VM clock drift that invalidates JWT nbf and exp claims.
Upstream IdP Key Rotation: Detects when Pinniped cannot fetch new public keys following IdP certificate renewal.
Brute Force Detection: Flags abnormal spikes in rejected bearer tokens indicating credential abuse attempts.

Inputs

Name	Type	Default	Description
`supervisor_metrics_url`	STRING	`http://pinniped-supervisor.pinniped-supervisor.svc.cluster.local:8080`	Internal HTTP URL exposing Prometheus metrics for Pinniped.
`target_jwt_issuer`	STRING	`https://auth.company.internal`	Upstream OIDC identity provider issuer URL monitored.
`max_clock_skew_seconds_threshold`	INT	`60`	Maximum allowable clock skew in seconds before alerting.
`max_token_rejection_count`	INT	`5`	Maximum allowable rejected JWT tokens before alerting.
`alert_on_jwks_refresh_failure`	BOOL	`true`	Toggle whether JWKS key set sync errors trigger an alert.

Expected outputs

{{ outputs.scan_pinniped_telemetry.vars.clock_skew_seconds }}: Measured clock skew drift in seconds against the issuer.
{{ outputs.scan_pinniped_telemetry.vars.token_rejections_total }}: Total count of rejected bearer authentication tokens.
{{ outputs.scan_pinniped_telemetry.vars.jwks_sync_errors_total }}: Count of JWKS public key retrieval failures.
{{ outputs.scan_pinniped_telemetry.vars.is_alerting }}: Boolean flag indicating if identity bounds are breached.
{{ outputs.scan_pinniped_telemetry.vars.audit_timestamp }}: ISO-8601 UTC timestamp of the Pinniped audit run.

Prerequisites

Pinniped Supervisor installed with Prometheus metrics enabled on port 8080.
Network connectivity from the Kestra worker to the Pinniped Supervisor service.
Configured Slack incoming webhook URL with permissions to post in IAM SecOps alert channels.

Secrets

SLACK_WEBHOOK_URL: Secure webhook URL for routing IAM alerts to the SecOps Slack channel.

Quick start

Verify that Pinniped Supervisor exposes Prometheus metrics on HTTP port 8080.
Add the SLACK_WEBHOOK_URL secret within your Kestra namespace settings.
Specify your target identity provider issuer URL in target_jwt_issuer.
Run a manual flow execution via the Kestra UI to validate metric ingestion.
Enable the periodic_pinniped_audit schedule trigger to establish automated monitoring.

Common pitfalls and troubleshooting

Issuer Claim Mismatch: If iss claim in incoming JWT does not match issuer exactly, tokens are discarded.
Egress Proxy TLS Interception: Corporate firewalls re-signing HTTPS traffic break JWKS signature validation.
Audience Validation Error: Tokens lacking configured aud audience claim fail authentication silently.
NTP Daemon Stoppage: Worker nodes without chronyd or systemd-timesyncd drift over time, causing skew.

How to extend

Automated JWKS Cache Flush: Trigger Kestra task to restart supervisor pods to force immediate JWKS cache reload.
PagerDuty Escalation: Escalate to on-call IAM engineers if token rejections exceed 50 in 15 minutes.
Multi-IdP Fleet Scan: Expand script parameters to monitor all declared JWTAuthenticator resources.

Pinniped Supervisor JWT Authenticator Clock Skew Sentinel

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Pinniped Supervisor JWT Authenticator Clock Skew Sentinel

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

More Cloud Blueprints

New to Kestra?