Broadcast Open Data Drops on X After the Pipeline Succeeds

The dataset landed in the bucket, the pipeline went green, and the announcement went out with a link that 404s because publishing and announcing were two separate rituals. This blueprint chains them. A stand-in io.kestra.plugin.core.http.Request represents the pipeline's publication step and requests the public dataset URL, so the announcement only runs when the link resolves. io.kestra.plugin.x.XExecution then posts the drop notification. The task is an execution notification poster, so the post always leads with the execution state and flow id, carries the announcement line from customMessage, and ends with a link to the execution, which makes the account a public, verifiable record that the drop pipeline ran. Discord receives an internal note with the period that went public. Because posts on X are public and instant, the announcement line is assembled from reviewed inputs, and the 280 character cap covers the full rendered post, status line and link included.

How it works

1. verify_dataset (io.kestra.plugin.core.http.Request) stands in for the data pipeline's publish step and requests the dataset URL. A non-success response fails the task, and nothing is announced for a link that does not resolve. Replace it with your real publication task.
2. post_announcement (io.kestra.plugin.x.XExecution) authenticates with the OAuth 1.0a credential set, consumer key and secret plus access token and secret, and posts the execution notification. The task renders its bundled execution template, so the post opens with the execution state and flow id, carries the customMessage announcement line, and closes with a link to the execution. The 280 character cap applies to that full rendered post, which keeps the custom line short by design.
3. notify_internal (io.kestra.plugin.discord.DiscordIncomingWebhook) posts the dataset name, period, and execution id to the team channel, so the data team knows the announcement is live.
4. monthly_drop (io.kestra.plugin.core.trigger.Schedule) fires on the first of the month at 09:00 UTC once enabled, matching a monthly publication cadence. It ships disabled so nothing posts before the first review.
5. The errors block sends a distinct Discord alert when either verification or posting fails, because a dataset that is published but unannounced reaches nobody.

What you get

• Announcements that fire only after the pipeline step succeeds, so success is broadcast and failure stays internal.
• Posts that carry the execution state and an execution link alongside the announcement line, public proof that the drop pipeline ran.
• A link check before every public post, no announced URL that 404s.
• An internal Discord note carrying the public copy's period, so the team sees what followers see.
• A failure path that reports when the post did not go out, separating published from announced.

Who it's for

• Open data teams who publish recurring dataset releases and announce each drop publicly.
• Data engineers who want the announcement to be the last task of the pipeline, not a calendar reminder.
• Anyone who has announced a data release before the file finished uploading.

Why orchestrate this with Kestra

A pipeline can end with a curl call to the X API, but that call cannot verify the artifact first, keep an audit trail of every public post, or alert a channel when publishing fails. Kestra sequences verification before announcement, keeps the four OAuth credentials in secrets, gives every public post an execution history, and turns a failed broadcast into a loud Discord message through the errors block.

Prerequisites

• An X developer app with OAuth 1.0a user context credentials for the account that posts.
• A publicly reachable dataset URL, or a real publication task to swap in for the stand-in.
• A Discord incoming webhook for internal notes and failure alerts.

Secrets

• X_CONSUMER_KEY: X app consumer key (API key).
• X_CONSUMER_SECRET: X app consumer secret (API secret).
• X_ACCESS_TOKEN: OAuth 1.0a access token for the posting account.
• X_ACCESS_SECRET: OAuth 1.0a access secret for the posting account.
• DISCORD_WEBHOOK_URL: Discord incoming webhook for internal notes and failure alerts.

Quick start

1. Add the five secrets to your Kestra namespace.
2. Execute the flow manually with the default inputs and confirm the post appears on the X account and the Discord note arrives, then delete the test post.
3. Replace verify_dataset with your real publication task, or point dataset_url at the real public location.
4. Review the post copy, update dataset_period, and set disabled: false on the monthly_drop trigger.

How to extend

• Switch to app-only authentication by replacing the four OAuth properties with bearerToken, as shown in the plugin documentation.
• Replace the stand-in with your real pipeline tasks, an upload to object storage followed by the same HTTP verification keeps the link check.
• Pull row counts or file sizes from the pipeline outputs and add them as customFields entries, each renders as its own key-value line in the post, within the same 280 character cap.
• Chain this flow after the pipeline flow with a io.kestra.plugin.core.trigger.Flow trigger instead of the schedule, so the broadcast follows every successful run.

Links

• X plugin documentation
• XExecution task reference