Schedule icon
Webhook icon
Assert icon
Request icon
Script icon
Set icon
If icon
SlackIncomingWebhook icon
Pause icon
Loop icon
Log icon

Rotate Stale Apache Polaris Service Principal Credentials After Approval

Detect stale Apache Polaris principal credentials, track rotation history in KV, and rotate with operator approval in Kestra without exposing the new secret.

Categories
DataInfrastructure

Apache Polaris manages service principals, each with an OAuth2 client ID and secret used to authenticate against the Iceberg REST catalog. Like any long-lived service credential, a Polaris principal's secret should be rotated periodically: the longer a credential lives unrotated, the larger the exposure window if it ever leaks into a log, a config file committed to source control, or a compromised CI runner. Polaris itself has no built-in alert for credential age, and nothing paginates an operator when a service account's secret has quietly been in place for a year.

This blueprint provides an automated guard that lists Polaris principals, evaluates each one's credential age against historical rotation state tracked in Kestra KV (deliberately not against Polaris's own lastUpdateTimestamp, which reflects any metadata change to the principal, not specifically a credential rotation), requests human approval, calls Polaris's rotateCredentials API, and confirms success by HTTP status code alone. The newly generated secret returned by that call is never referenced in any log message, Slack notification, or KV write this flow produces.

How it works

  1. Pre-flight Assertion: validate_inputs (io.kestra.plugin.core.execution.Assert) checks that the age threshold and the per-run rotation cap are both positive.
  2. Principal Discovery: list_principals (io.kestra.plugin.core.http.Request) calls GET /api/management/v1/principals using a Bearer token supplied as input, since Polaris's built-in /v1/oauth/tokens endpoint is documented as deprecated for removal and this blueprint intentionally does not implement that exchange itself.
  3. Triage: triage_principals (io.kestra.plugin.scripts.python.Script) computes each principal's credential age from this flow's own KV-tracked rotation history (inputs.cursor_key), falling back to Polaris's createTimestamp for a principal never rotated by this flow before. Principals in exclude_principals (for example a bootstrap or break-glass account) are always skipped. Candidates beyond max_rotations_per_run are deferred to a later run, prioritizing the oldest credentials first.
  4. State Persistence: save_tracker_kv (io.kestra.plugin.core.kv.Set) writes the baseline per-principal state to Kestra KV.
  5. Slack Triage Alert: When stale principals exist, dispatch_slack_alert posts the affected principal names and credential ages, never any credential material.
  6. Human Approval Gate: When auto_remediate is enabled and dry_run is false, pause_for_operator_approval (io.kestra.plugin.core.flow.Pause) halts execution until an operator resumes with approved: true.
  7. Rotate and Confirm: rotate_stale_principals (io.kestra.plugin.core.flow.Loop) calls POST /api/management/v1/principals/{name}/rotate per approved principal, then assert_rotation_succeeded (io.kestra.plugin.core.execution.Assert) confirms only the HTTP status code, deliberately never parsing the response body that carries the new secret.
  8. Rotation State Update: mark_rotated_in_kv (io.kestra.plugin.scripts.python.Script) merges the new rotation timestamp for just the rotated principals into the KV snapshot, leaving every other principal's tracked state untouched, and save_rotated_tracker_kv persists it.
  9. Audit & Resolution: The audit manifest (principal names and ages only) is archived in KV, and a resolution card is posted to Slack noting that Polaris retains the prior credential as a secondary secret, so this is a graceful rotation rather than an instant cutover.
  10. Failure Handling: A flow-level errors block alerts Slack if Polaris connectivity, bearer token validity, or scripting fails unexpectedly.

What you get

  • Automated detection of Polaris service principal credentials exceeding your organization's rotation policy.
  • Rotation history tracked independently of Polaris's own metadata timestamp, which is not rotation-specific.
  • Human-in-the-loop approval before any credential is rotated.
  • A hard guarantee, by construction, that the newly generated secret is never written to a log, a Slack message, or Kestra KV.
  • A per-run rotation cap that prioritizes the oldest credential first instead of rotating a whole fleet of principals at once.

Who it's for

  • Platform and Security Engineers operating Apache Polaris as an Iceberg REST catalog for a data lake.
  • Teams with a credential rotation policy (for example, a 90-day maximum age) who need to enforce it without a manual quarterly audit.
  • Compliance and audit teams who need a record of when each service principal's credentials were last rotated.

Why orchestrate this with Kestra

Polaris has no built-in credential-age alerting, and nothing distinguishes "this principal was just updated" from "this principal's credentials were just rotated" in its own API, that distinction has to be tracked externally. Kestra adds declarative scheduling, KV-backed rotation history independent of Polaris's own timestamps, an interactive approval gate, and auditable rotation, all while structurally guaranteeing the new secret never leaks into an execution log.

Prerequisites

  • A reachable Apache Polaris server (recent release with the management API) with its HTTP port (default 8181) reachable from the Kestra worker.
  • A valid OAuth2 bearer token for a Polaris principal with permission to list and rotate other principals' credentials, obtained via your own identity provider configured against Polaris's oauth2-server-uri.
  • A Slack Incoming Webhook URL for triage and resolution notifications.

Secrets

  • POLARIS_BEARER_TOKEN: A valid OAuth2 bearer token for the Polaris management API.
  • SLACK_WEBHOOK_URL: Slack Incoming Webhook endpoint for triage and resolution alerts.
  • POLARIS_GUARD_WEBHOOK_KEY: Secret authentication token protecting the on-demand webhook trigger.

Quick start

  1. Configure POLARIS_BEARER_TOKEN, SLACK_WEBHOOK_URL, and POLARIS_GUARD_WEBHOOK_KEY in your Kestra namespace secrets.
  2. Import this blueprint and set polaris_url to your Polaris server's endpoint, and exclude_principals to any bootstrap or break-glass principal names.
  3. Run a manual execution with the default dry_run: true to audit current credential ages without mutating anything.
  4. Review the execution logs, Slack triage card, and the KV entry under inputs.cursor_key.
  5. Run with auto_remediate: true and dry_run: false to arm remediation; when stale principals are detected, coordinate with the consuming service's owners, then approve the pause_for_operator_approval task with approved: true.
  6. Enable the periodic_credential_age_audit schedule trigger for continuous monitoring.

How to extend

  • Downstream Secret Sync: After rotation, add a task that pushes the new credentials into your secrets manager (for example Vault or AWS Secrets Manager) directly from the rotate_credentials response, instead of surfacing them manually, keeping the no-logging guarantee intact by only ever piping the response into the secret store task's input.
  • Reset Instead of Rotate: For a principal whose credentials are suspected compromised rather than simply old, swap rotate_credentials for a call to POST /principals/{name}/reset, which invalidates the prior secondary secret immediately instead of preserving a grace period.
  • PagerDuty Escalation: Add io.kestra.plugin.pagerduty.PagerDutyAlert when a principal's credential age exceeds a second, much higher critical threshold, signaling a rotation policy that has been silently ignored for a long time.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.