Schedule icon
Webhook icon
Assert icon
Script icon
Loop icon
Set icon
If icon
SlackIncomingWebhook icon
Fail icon
Log icon
OutputValues icon

Feast Feature Store Offline-to-Online Sync Drift & Parity Guard

Detect Feast offline-to-online feature store synchronization drift and parity skew across DuckDB, PostgreSQL, Redis, and DynamoDB in Kestra.

Categories
AIData

Production machine learning systems depend on real-time feature stores such as Feast to serve sub-millisecond predictions. While offline data lakes (DuckDB, Snowflake, BigQuery, PostgreSQL) store historical feature tables for model training, online key-value stores (Redis, DynamoDB) cache the latest values for low-latency inference.

When Feast materialization jobs stall, lag, or silently fail, online feature caches become stale, and values diverge from offline gold records. This training-serving skew degrades model inference accuracy without throwing fatal application errors.

This blueprint provides an automated parity and drift guard that audits Feast feature views, measures materialization staleness, samples entity keys to compare offline historical values against online cached values, persists drift metrics into Kestra's KV store, and alerts ML engineering on Slack with actionable diagnostics.

How it works

  1. Pre-flight Assertion: Validates operational input parameters, verifying positive sample sizes, reasonable staleness thresholds, and non-empty feature view lists.
  2. Feast Metadata & Timestamp Audit: Inspects recent materialization timestamps per feature view from the Feast registry or store metadata.
  3. Staleness Duration Calculation: Computes current_time - last_materialized_at and evaluates whether synchronization has exceeded max_staleness_threshold_hours.
  4. Stratified Parity Comparison: Samples random entity keys across configured feature views, retrieving corresponding offline values and online cache entries to identify value mismatches and missing keys.
  5. Drift Metric Computation: Calculates the parity mismatch percentage per feature view and classifies synchronization state (HEALTHY_IN_SYNC, WARNING_STALE_MATERIALIZATION, CRITICAL_PARITY_BREACH, or CRITICAL_DRIFT_AND_STALE).
  6. Per-View Kestra KV Storage: Loops over feature view evaluations using io.kestra.plugin.core.flow.Loop and persists structured sync metrics to feast_feature_drift_{feature_view}.
  7. Audit Manifest Recording: Saves an aggregated system evaluation manifest under key feast_feature_drift_summary in the Kestra namespace KV store.
  8. Actionable Slack Alerts: Formats and delivers prioritized Slack notifications with affected feature views, entity keys, and drift percentages.
  9. Policy Gate Enforcement: Optionally triggers workflow failure via io.kestra.plugin.core.execution.Fail when fail_on_parity_breach is enabled to prevent deploying models against drifted feature stores.

What you get

  • Continuous verification of offline-to-online feature store consistency to eliminate training-serving skew.
  • Proactive detection of stalled Feast materialization jobs before model inference degrades.
  • Namespace Key-Value store audit history per feature view under feast_feature_drift_{feature_view}.
  • Line-item diagnostics for specific entity keys exhibiting cache drift or missing online keys.
  • Multi-store adaptability supporting DuckDB, PostgreSQL, BigQuery, Snowflake, Redis, and DynamoDB.

Who it's for

  • Machine Learning Engineers (MLEs) and MLOps teams operating Feast in production.
  • Data Engineers maintaining offline batch pipelines and online cache materialization schedules.
  • Platform and Reliability Engineers monitoring ML serving infrastructure data consistency.

Why orchestrate this with Kestra

Feast delegates feature store orchestration and materialization triggering to external schedulers. Running unmonitored materialization crons introduces blind spots when upstream ETL jobs are delayed or network partitions prevent cache writes. Orchestrating this parity guard with Kestra provides declarative scheduling, webhook triggering after ETL completion, built-in KV persistence for downstream model canary checks, secrets security, and multi-channel alerting without writing bespoke daemon microservices.

Prerequisites

  • A Feast feature repository or access to your offline store (DuckDB, PostgreSQL, BigQuery, Snowflake) and online store (Redis, DynamoDB, SQLite).
  • Network connectivity from the Kestra worker to the target feature store endpoints.
  • A Slack Incoming Webhook URL to receive drift notifications.

Secrets

  • SLACK_WEBHOOK_URL: Slack Incoming Webhook endpoint URL used for drift and failure notifications.
  • FEAST_GUARD_WEBHOOK_KEY: Secret authentication token protecting the on-demand webhook trigger.
  • FEAST_REDIS_HOST: (Optional) Hostname or IP address for the Redis online feature store.
  • FEAST_REDIS_PORT: (Optional) Port number for Redis (defaults to 6379).

Quick start

  1. Configure SLACK_WEBHOOK_URL and FEAST_GUARD_WEBHOOK_KEY in your Kestra namespace secrets.
  2. Import this blueprint and set offline_store_type and online_store_type to match your infrastructure.
  3. Specify your production feature views in feature_views (e.g. ["user_engagement_features", "credit_scoring_features"]).
  4. Execute the flow manually to perform a baseline parity and staleness audit.
  5. Enable the periodic_sync_guard schedule trigger to run automated drift audits every 2 hours.

How to extend

  • Automated Cache Re-materialization: Connect an io.kestra.plugin.core.flow.Subflow or CLI task to automatically trigger feast materialize-incremental when staleness is detected.
  • PagerDuty Incident Escalation: Integrate io.kestra.plugin.pagerduty.notifications.PagerDutyAlert to trigger high-severity on-call pages for critical parity breaches.
  • Pre-Deployment Model Canary Gate: Read feast_feature_drift_{feature_view} using io.kestra.plugin.core.kv.Get in ML model deployment pipelines to block releases when feature drift is detected.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.