Snowflake Automatic Clustering Cost and FinOps Guard

In Snowflake, Automatic Clustering maintains optimal micro-partition sorting for multi-terabyte tables without manual vacuuming or defragmentation. However, because reclustering executes as a continuous serverless background service, poorly chosen cluster keys on frequently updated, ingested, or deleted tables cause Snowflake to endlessly rewrite micro-partitions.

This continuous churn can consume hundreds of compute credits every day without generating any user-visible query errors, quietly burning thousands of dollars of monthly cloud budget.

This blueprint establishes an automated FinOps guardrail for Snowflake background compute. Executing daily or on-demand, it inspects SNOWFLAKE.ACCOUNT_USAGE.AUTOMATIC_CLUSTERING_HISTORY, aggregates credits consumed per table, and flags tables exceeding your credit threshold. When runaway clustering is detected, it dispatches an actionable Slack alert with dollar cost estimates, reclustered gigabytes, and immediate SUSPEND RECLUSTER remediation commands.

How it works

1. Scheduled FinOps Audit: The daily_clustering_audit trigger (io.kestra.plugin.core.trigger.Schedule) executes daily at 08:00 UTC or on-demand via the Kestra UI.
2. Clustering Spend Query: The audit_clustering_spend task (io.kestra.plugin.jdbc.snowflake.Query) queries snowflake.account_usage.automatic_clustering_history, grouping credits used and reclustered bytes by table.
3. Evaluation Gate: The evaluate_clustering_spikes flowable task (io.kestra.plugin.core.flow.If) branches based on whether any table exceeded max_credits_per_table.
4. Slack Alert Dispatch: If runaway clustering is detected, notify_finops_slack (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) delivers a diagnostic card detailing the table name, credits burned, dollar costs, and remediation guidance.
5. Nominal Logging: When all tables remain within normal credit thresholds, log_nominal_clustering records compliant status in execution logs.
6. Audit Manifest: The export_clustering_manifest task records telemetry outputs for cloud budget dashboards.

What you get

• Early warning on runaway serverless background credit consumption.
• Identification of tables with inefficient or overly granular clustering keys.
• Automated dollar cost estimation directly in Slack notifications.
• Zero human overhead for continuous Snowflake serverless compute governance.

Who it is for

• FinOps Practitioners managing cloud data warehouse compute commitments.
• Snowflake Data Platform Engineers designing clustering keys for large analytical tables.
• Analytics Engineering Leads monitoring dbt incremental model clustering performance.

Why orchestrate this with Kestra

Snowflake resource monitors only apply to virtual warehouses and cannot automatically cap serverless services like Automatic Clustering. Kestra provides declarative, serverless orchestration: it connects securely via JDBC, inspects serverless history daily, evaluates credit anomalies, and delivers rich alerts directly to your team on Slack.

Inputs

Name	Type	Default	Description
lookback_days	INT	1	Number of days of historical automatic clustering activity to audit.
max_credits_per_table	FLOAT	10.0	Daily credit threshold beyond which table automatic clustering triggers an alert.
credit_price_usd	FLOAT	3.00	Estimated dollar cost per Snowflake compute credit.
slack_channel	STRING	#finops-alerts	Slack channel destination for clustering cost alerts.

Expected outputs

• {{ outputs.audit_clustering_spend.rows }}: Array of table records containing database_name, schema_name, table_name, total_credits_consumed, estimated_cost_usd, and reclustered_gigabytes.
• {{ outputs.audit_clustering_spend.size }}: Total count of tables breaching the daily clustering credit threshold.
• {{ outputs.evaluate_clustering_spikes }}: Result of conditional branch evaluation.
• {{ outputs.export_clustering_manifest.value }}: Structured JSON telemetry manifest recording execution timestamp and status.

Prerequisites

• A Snowflake account with access to SNOWFLAKE.ACCOUNT_USAGE schema views.
• A Snowflake service account with a role granted IMPORTED PRIVILEGES on the SNOWFLAKE database.
• A Slack Incoming Webhook URL targeting your FinOps notification channel.

Secrets

• SNOWFLAKE_JDBC_URL: Snowflake JDBC connection string (e.g. jdbc:snowflake://<account_locator>.snowflakecomputing.com?warehouse=COMPUTE_WH&db=SNOWFLAKE&schema=ACCOUNT_USAGE).
• SNOWFLAKE_USER: Username of the service account authorized to query account usage.
• SNOWFLAKE_PASSWORD: Password for the Snowflake service account.
• SLACK_WEBHOOK_URL: Slack Incoming Webhook endpoint URL used for sending alert notifications.

Quick start

1. Configure SNOWFLAKE_JDBC_URL, SNOWFLAKE_USER, SNOWFLAKE_PASSWORD, and SLACK_WEBHOOK_URL in your Kestra namespace secrets.
2. Import this flow YAML into your Kestra instance.
3. Click Execute in the UI to run an initial audit of recent clustering activity.
4. Review execution outputs to inspect table credit consumption and reclustered data volumes.

Common pitfalls and troubleshooting

• Account Usage Latency: Views in SNOWFLAKE.ACCOUNT_USAGE such as AUTOMATIC_CLUSTERING_HISTORY experience up to 45 to 120 minutes of data latency. Schedule the flow daily for fully reconciled metrics.
• Clustering Key Cardinality: Tables clustered on high-cardinality keys (like timestamps or unique IDs) cause excessive micro-partition splitting. Consider clustering on coarser expressions like to_date(timestamp).
• Suspending Reclustering: When an alert fires, you can immediately suspend reclustering using ALTER TABLE <table_name> SUSPEND RECLUSTER; while optimizing cluster keys.

How to extend

• Add an automated remediation step executing ALTER TABLE ... SUSPEND RECLUSTER; directly if spend exceeds a critical emergency threshold.
• Query SYSTEM$CLUSTERING_INFORMATION to compute average clustering depth and partition overlap metrics.
• Email weekly reclustering cost reports to data engineering managers.

Links

• Kestra Snowflake Plugin Documentation: https://kestra.io/plugins/plugin-jdbc-snowflake
• Kestra Slack Plugin Documentation: https://kestra.io/plugins/plugin-notifications
• Snowflake Automatic Clustering Documentation: https://docs.snowflake.com/en/user-guide/tables-auto-reclustering