Huawei OBS Stale Object Retention Sentinel with Kestra

id: huawei-obs-stale-object-retention-sentinel
namespace: company.team

description: |
  Scan a Huawei OBS bucket prefix for objects past their retention age, alert Slack with the
  reclaimable byte count, and optionally delete only the stale objects, one key at a time, then
  re-check and verify none remain. Off by default: nothing is deleted unless both auto_cleanup is
  true and dry_run is false.

inputs:
  - id: region
    type: STRING
    displayName: Huawei Cloud region
    description: Region identifier used to sign OBS requests, for example
      eu-west-101 or cn-north-4.
    defaults: "eu-west-101"

  - id: bucket
    type: STRING
    displayName: OBS bucket name
    description: The bucket to scan. It must already exist.
    defaults: "<your-obs-bucket>"

  - id: prefix
    type: STRING
    displayName: Object key prefix
    description: Only objects under this prefix are scanned, for example tmp/ or
      staging/exports/.
    defaults: "tmp/"

  - id: regexp
    type: STRING
    displayName: Client-side key filter
    description: Optional regular expression objects must also match, applied after
      the prefix. Leave empty to match every object under the prefix.
    defaults: ""

  - id: retention_days
    type: INT
    displayName: Retention age in days
    description: Objects under the prefix with a last-modified date older than this
      many days are reported as stale.
    defaults: 30

  - id: auto_cleanup
    type: BOOL
    displayName: Auto cleanup
    description: Allow deleting the stale objects found this run.
    defaults: false

  - id: dry_run
    type: BOOL
    displayName: Dry run
    description: Must be false, together with auto_cleanup, for Delete to run.
    defaults: true

tasks:
  - id: list_stale_candidates
    type: io.kestra.plugin.huawei.obs.List
    description: Read-only. Lists every object under the prefix; staleness is judged
      afterwards from lastModified.
    accessKeyId: "{{ secret('HUAWEI_AK') }}"
    secretAccessKey: "{{ secret('HUAWEI_SK') }}"
    region: "{{ inputs.region }}"
    bucket: "{{ inputs.bucket }}"
    prefix: "{{ inputs.prefix }}"
    regexp: "{{ inputs.regexp != '' ? inputs.regexp : null }}"

  - id: evaluate
    type: io.kestra.plugin.core.output.OutputValues
    description: Classifies every listed object against retention_days. lastModified
      is compared by its date portion, which is enough for a day-granularity
      retention check.
    values:
      stale_objects: "{{ {'o': outputs.list_stale_candidates.objects, 'cutoff': (now()
        | dateAdd(-inputs.retention_days, 'DAYS') | date('yyyy-MM-dd'))} |
        jq('.cutoff as $c | (.o // []) | map(select((.lastModified[0:10]) <
        $c))') | first | toJson }}"
      stale_count: "{{ {'o': outputs.list_stale_candidates.objects, 'cutoff': (now() |
        dateAdd(-inputs.retention_days, 'DAYS') | date('yyyy-MM-dd'))} |
        jq('.cutoff as $c | (.o // []) | map(select((.lastModified[0:10]) < $c))
        | length') | first }}"
      stale_size_bytes: "{{ {'o': outputs.list_stale_candidates.objects, 'cutoff':
        (now() | dateAdd(-inputs.retention_days, 'DAYS') | date('yyyy-MM-dd'))}
        | jq('.cutoff as $c | (.o // []) | map(select((.lastModified[0:10]) <
        $c)) | map(.size) | add // 0') | first }}"
      severity: "{{ (({'o': outputs.list_stale_candidates.objects, 'cutoff': (now() |
        dateAdd(-inputs.retention_days, 'DAYS') | date('yyyy-MM-dd'))} |
        jq('.cutoff as $c | (.o // []) | map(select((.lastModified[0:10]) < $c))
        | length') | first)) > 0 ? 'STALE_OBJECTS_FOUND' : 'OK' }}"

  - id: route
    type: io.kestra.plugin.core.flow.If
    condition: "{{ outputs.evaluate.values.severity != 'OK' }}"
    then:
      - id: alert
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {{ {'text': 'OBS retention: ' ~ outputs.evaluate.values.stale_count ~ ' object(s) under obs://' ~ inputs.bucket ~ '/' ~ inputs.prefix ~ ' are older than ' ~ inputs.retention_days ~ ' days, ' ~ outputs.evaluate.values.stale_size_bytes ~ ' bytes reclaimable.' ~ (inputs.auto_cleanup and not inputs.dry_run ? ' Deleting them now.' : ' No change applied (auto_cleanup=' ~ inputs.auto_cleanup ~ ', dry_run=' ~ inputs.dry_run ~ ').')} | toJson }}

      - id: remediate_gate
        type: io.kestra.plugin.core.flow.If
        condition: "{{ inputs.auto_cleanup and not inputs.dry_run and
          outputs.evaluate.values.stale_count > 0 }}"
        then:
          - id: cleanup
            type: io.kestra.plugin.core.flow.Loop
            concurrencyLimit: 1
            values: "{{ outputs.evaluate.values.stale_objects }}"
            tasks:
              - id: delete_object
                type: io.kestra.plugin.huawei.obs.Delete
                accessKeyId: "{{ secret('HUAWEI_AK') }}"
                secretAccessKey: "{{ secret('HUAWEI_SK') }}"
                region: "{{ inputs.region }}"
                bucket: "{{ inputs.bucket }}"
                key: "{{ item.value | jq('.key') | first }}"

          - id: reread
            type: io.kestra.plugin.huawei.obs.List
            accessKeyId: "{{ secret('HUAWEI_AK') }}"
            secretAccessKey: "{{ secret('HUAWEI_SK') }}"
            region: "{{ inputs.region }}"
            bucket: "{{ inputs.bucket }}"
            prefix: "{{ inputs.prefix }}"
            regexp: "{{ inputs.regexp != '' ? inputs.regexp : null }}"

          - id: verify_result
            type: io.kestra.plugin.core.flow.If
            description: Every object deleted in cleanup must be gone from the reread
              listing.
            condition: "{{ ({'o': outputs.reread.objects, 'cutoff': (now() |
              dateAdd(-inputs.retention_days, 'DAYS') | date('yyyy-MM-dd'))} |
              jq('.cutoff as $c | (.o // []) | map(select((.lastModified[0:10])
              < $c)) | length') | first) > 0 }}"
            then:
              - id: remediation_failed
                type: io.kestra.plugin.core.execution.Fail
                errorMessage: "Delete ran, but stale objects are still present under the prefix
                  after cleanup."
            else:
              - id: remediated
                type: io.kestra.plugin.core.log.Log
                message: "Deleted {{ outputs.evaluate.values.stale_count }} stale object(s) ({{
                  outputs.evaluate.values.stale_size_bytes }} bytes) under
                  obs://{{ inputs.bucket }}/{{ inputs.prefix }} and verified
                  none remain."
    else:
      - id: compliant
        type: io.kestra.plugin.core.log.Log
        message: "No object under obs://{{ inputs.bucket }}/{{ inputs.prefix }} is older
          than {{ inputs.retention_days }} days."

  - id: summary
    type: io.kestra.plugin.core.log.Log
    message: "OBS retention sentinel complete for obs://{{ inputs.bucket }}/{{
      inputs.prefix }}: {{ outputs.evaluate.values.stale_count }} stale
      object(s), {{ outputs.evaluate.values.stale_size_bytes }} bytes, severity
      {{ outputs.evaluate.values.severity }}."

errors:
  - id: alert_flow_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {{ {'text': 'Huawei OBS stale object retention sentinel failed (execution ' ~ execution.id ~ '). Check the logs.'} | toJson }}

triggers:
  - id: schedule_daily_at_0230
    type: io.kestra.plugin.core.trigger.Schedule
    description: Shipped disabled, and always in safe mode.
    cron: "30 2 * * *"
    disabled: true
    inputs:
      region: "eu-west-101"
      bucket: "<your-obs-bucket>"
      prefix: "tmp/"
      regexp: ""
      retention_days: 30
      auto_cleanup: false
      dry_run: true

Temporary exports, staging landing zones and old job output tend to accumulate in object storage because nothing ever looks back at them. Huawei OBS has lifecycle rules, but they are bucket-wide or prefix-wide console configuration with no review step and no record of what they actually deleted on a given day. This blueprint does the same job as a flow: list what's under a prefix, decide what's stale from lastModified, alert on what it found, and, only when you ask it to, delete exactly those objects and prove they are gone.

This blueprint was created by Mohit Jeswani.

How it works

list_stale_candidates (io.kestra.plugin.huawei.obs.List) lists every object under prefix in bucket, optionally narrowed further by regexp. Read-only.
evaluate (io.kestra.plugin.core.output.OutputValues) compares each object's lastModified date against now() - retention_days and produces the stale object list, their count, their total size in bytes, and a severity.
route (io.kestra.plugin.core.flow.If):
- compliant: logs that nothing is past retention.
- alert: posts the stale count and reclaimable bytes to Slack.
  - remediate_gate runs only when auto_cleanup is true and dry_run is false.
  - cleanup (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) calls io.kestra.plugin.huawei.obs.Delete once per stale key, never the whole prefix, so an object that is not actually stale is never touched.
  - reread lists the prefix again, and verify_result fails the flow if any object past retention_days is still there.
summary logs the stale count, reclaimable bytes and severity, closing the governance cycle.
errors alerts Slack if the flow itself fails, for example on an expired AK/SK.

What you get

A Slack alert with a number, not a guess, the exact count and byte total of what is sitting past retention under a prefix.
Deletion that only ever removes the objects the same run identified as stale, never a prefix-wide wipe.
A verified record, in the execution's own logs, that cleanup actually emptied out what it claimed to.

Who it's for

Platform and data teams who land temporary exports or staging files in Huawei OBS and have no process catching the ones nobody cleaned up.
Teams evaluating the Huawei Cloud plugin who want a worked example of obs.List and obs.Delete used together safely.
FinOps-minded teams who want object storage cost creep visible before the monthly bill, not after.

Why orchestrate this with Kestra

An OBS lifecycle rule is silent: it runs in the background with no alert, no count of what it removed, and no easy way to preview the impact before enabling it. Kestra keeps retention_days and the prefix as flow inputs, reports what it found before anything is deleted, requires two separate flags to actually delete, and leaves a full execution history of every run, compliant or not, remediated or not.

Prerequisites

A Huawei Cloud account with OBS enabled and a bucket already created.
An AK/SK pair (IAM access key and secret key) with OBS read permission, and delete permission on the bucket if auto_cleanup is ever enabled.
A Slack incoming webhook URL.

This flow calls the Huawei Cloud OBS API directly over HTTPS; there is no official local Huawei Cloud emulator. OBS is S3-API compatible, so List and Delete can be pointed at a local MinIO container via endpointOverride and pathStyleAccess: true for a dry run of the logic, but the actual lastModified ages used to prove retention_days need real aged objects, which is easiest against a real trial-tier OBS bucket.

Secrets

HUAWEI_AK: the IAM access key used by every OBS task in this flow. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_HUAWEI_AK.
HUAWEI_SK: the IAM secret key paired with HUAWEI_AK. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_HUAWEI_SK.
SLACK_WEBHOOK_URL: used by alert and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Type	Default	Purpose
`region`	STRING	`eu-west-101`	Huawei Cloud region used to sign OBS requests
`bucket`	STRING	`<your-obs-bucket>`	Bucket to scan
`prefix`	STRING	`tmp/`	Only objects under this prefix are scanned
`regexp`	STRING	empty	Optional extra key filter, applied after the prefix
`retention_days`	INT	`30`	Age past which an object is reported stale
`auto_cleanup`	BOOL	`false`	Allow deleting the stale objects found this run
`dry_run`	BOOL	`true`	Must be `false` for Delete to run

Outputs

{{ outputs.list_stale_candidates.objects }}: every object under the prefix, each with key, size, lastModified, etag, owner.
{{ outputs.evaluate.values.stale_count }} / {{ outputs.evaluate.values.stale_size_bytes }}: how many objects are past retention and their combined size.
{{ outputs.evaluate.values.severity }}: OK or STALE_OBJECTS_FOUND.
{{ outputs.reread.objects }}: only present after a cleanup run, the post-delete listing used to verify.

Pitfalls

prefix is the entire blast radius for auto_cleanup. A prefix of "" scans and can delete the whole bucket; scope it narrowly.
Staleness is judged by lastModified, the last write time, not by when an object was created or first accessed. An object rewritten in place resets its own clock.
List iterates every page under the prefix into memory before evaluate runs; a prefix with millions of keys should be narrowed further or should set maxResults on list_stale_candidates directly.
The AK/SK pair needs OBS delete permission only if auto_cleanup is ever turned on; keep it read-only otherwise.
OBS versioned buckets do not free space on delete without a versionId: Delete places a delete marker instead of removing the object. Pass a lifecycle rule or a separate purge for versioned buckets.

Quick start

Add the HUAWEI_AK, HUAWEI_SK and SLACK_WEBHOOK_URL secrets.
Set bucket and prefix to a real location you expect to accumulate temporary objects.
Run the flow once with the defaults and read the stale count in summary.
Set auto_cleanup: true and dry_run: false on a non-production prefix first, confirm verify_result logs remediated, then apply the same change to production.
Enable the schedule_daily_at_0230 trigger, which always runs in safe mode until you change its mapped inputs.

How to extend

Narrow by owner or by a key pattern in regexp to scope cleanup to one team's exports.
Replace Delete in the cleanup loop with a copy-then-delete into a cheaper storage class if Huawei OBS exposes one for your bucket, instead of removing the object outright.
Track stale_size_bytes over time by logging it to a table with io.kestra.plugin.core.output.OutputValues on every scheduled run, compliant or not.
Point this at several prefixes with io.kestra.plugin.core.flow.Loop around the whole flow to cover a fleet of staging locations in one schedule.

Huawei OBS Stale Object Retention Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Inputs

Outputs

Pitfalls

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Huawei OBS Stale Object Retention Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Inputs

Outputs

Pitfalls

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Cloud Blueprints

Gate OIDC Signing-Key Rollovers with a Pinned JWKS

Multi-cloud deployment pipeline with Docker, ngrok, GCP Cloud Run, and Azure Container Apps

Prune noncurrent object versions billed forever behind a versioned bucket

New Relic APM Apdex Score Degradation Sentinel

Alert on nginx 5xx Error Rate

Memcached Slab Eviction Sentinel

More Infrastructure Blueprints

OPA-Gated IaC Applies with Human Approval

Huawei IAM Access Key Rotation Hygiene Sentinel

Apache Iceberg Partition Compaction and Lakehouse Maintenance Gate

HTTP Redirect Chain Audit

Huawei DataArts Factory Job Run Health Gate

Hugging Face Model Safetensors Security and Serialization Guard

New to Kestra?