Extract data, mask sensitive columns using DuckDB and load it to BigQuery
Extract CSV data, mask PII columns with DuckDB SQL, then load the anonymized result into BigQuery.
Fetch HackerNews top stories with Kestra, store them in PostgreSQL, generate an AI trend digest with GPT-4o-mini, and post a daily briefing to Slack.
Build a daily tech intelligence pipeline that pulls trending HackerNews stories, lands them in a PostgreSQL table for historical analysis, and broadcasts an AI-generated digest to your team. This blueprint solves the problem of turning a noisy public feed into a clean, deduplicated, queryable PostgreSQL dataset plus a human-readable summary, all on an automated daily schedule with no manual collection or copy-paste.
daily_digest trigger (io.kestra.plugin.core.trigger.Schedule, cron 0 8 * * *) starts the flow every morning at 08:00 UTC.fetch_story_ids (io.kestra.plugin.core.http.Request) calls the HackerNews Firebase API for the selected feed (topstories, newstories, or beststories).fetch_stories (io.kestra.plugin.core.flow.Loop) fans out across the top N story IDs with a concurrencyLimit of 5, fetching each story detail in parallel via a nested io.kestra.plugin.core.http.Request.summarize_stories (io.kestra.plugin.scripts.python.Script) parses and normalizes the raw Firebase responses, extracting id, title, url, score, author, comments, type, and timestamp, then sorts by score.summarize_with_ai (io.kestra.plugin.openai.ChatCompletion) sends the story list to gpt-4o-mini to write a concise 3 to 5 bullet trend digest.store_to_postgres (io.kestra.plugin.scripts.python.Script) upserts every story into a hackernews_stories table using INSERT ... ON CONFLICT DO UPDATE, keeping a clean historical archive.post_digest_to_slack (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) posts the AI digest and story count to a Slack channel.Loop with bounded concurrency.PostgreSQL has no built-in scheduler, retrieval, or alerting: it stores rows, nothing more. Kestra supplies the scheduled trigger, parallel HTTP retrieval, Python transformation, AI enrichment, and Slack delivery around the database, with retries, full execution lineage, and a declarative YAML definition you can version control. Event and schedule triggers, per-task observability, and replay close the orchestration gap that the database itself cannot fill.
gpt-4o-mini.OPENAI_API_KEYPOSTGRES_HOSTPOSTGRES_DBPOSTGRES_USERPOSTGRES_PASSWORDSLACK_WEBHOOK_URLstory_type and max_stories inputs if needed.