Commands icon
Docker icon
Script icon
If icon
SlackIncomingWebhook icon
Fail icon
Loop icon
Set icon
Pause icon

Migrate NATS JetStream Consumers to a Mirrored Stream with a Verified Cutover

Migrate NATS JetStream consumers to a mirrored stream, resuming at the exact ack-floor sequence, with approval and read-back verification.

Categories
Data

NATS JetStream's native stream mirroring keeps a target stream's messages in sync with a source stream, the same role MirrorMaker 2 plays for Kafka. But mirroring only copies the data: durable consumers still have to be recreated on the target, and if they are not pinned to the exact sequence each one had actually acknowledged on the source, a naive recreation either replays already-processed messages or skips ones that were never acknowledged. This blueprint reads each source consumer's real ack_floor sequence, refuses to proceed unless the target mirror has genuinely caught up, and recreates every consumer on the target at precisely the right starting point, then reads each one back to confirm it took before calling the cutover done.

How it works

  1. inspect_source_and_mirror (io.kestra.plugin.scripts.shell.Commands, official natsio/nats-box image) runs nats stream info --json against both clusters.
  2. evaluate_mirror_readiness (io.kestra.plugin.scripts.python.Script) refuses the cutover unless the target's mirror.name actually matches the source stream and mirror.lag is exactly 0 - confirmed from nats-server's own StreamSourceInfo struct.
  3. evaluate_safety_gate (io.kestra.plugin.core.flow.If) stops the entire flow with a Slack alert and io.kestra.plugin.core.execution.Fail before any consumer is touched if the mirror is not safely caught up.
  4. capture_consumer_positions (io.kestra.plugin.core.flow.Loop) reads each listed consumer's real ack_floor.stream_seq on the source - confirmed from nats-server's ConsumerInfo/SequenceInfo structs.
  5. save_migration_plan_kv (io.kestra.plugin.core.kv.Set) persists the full captured plan before anything on the target changes.
  6. pause_for_operator_approval (io.kestra.plugin.core.flow.Pause) waits indefinitely for a human to review the exact sequences before anything is created on the target.
  7. apply_cutover creates each consumer on the target mirror with nats consumer add ... --deliver <ack_floor + 1> - confirmed from natscli's own setStartPolicy source, where a plain numeric --deliver value sets DeliverPolicy=by_start_sequence and OptStartSeq to that exact number.
  8. verify_cutover reads every newly created target consumer back and compares its real config.opt_start_seq against what was requested, rather than trusting the create command's exit code.
  9. evaluate_verification_results alerts Slack with the specific failing consumer(s) if any verification failed, or confirms success with the verified count if all passed.
  10. The errors block alerts Slack separately if the flow itself fails outright.

What you get

  • A migration that never touches the source stream or its consumers - only reads from the source, only writes to the target.
  • An explicit, provable readiness gate (mirror.lag == 0, correct mirror source name) instead of assuming the mirror is caught up.
  • Gap-free, duplicate-free consumer cutover computed from each consumer's real acknowledged position, not a guessed or "start from the beginning" recreation.
  • A read-back verification step per consumer, naming exactly which one failed if the created consumer does not match the intended starting sequence.
  • A full KV audit record of the computed migration plan, saved before the target is touched.

Who it's for

  • Platform engineers migrating event-driven services between NATS clusters (datacenter move, cloud migration, major version upgrade) who cannot afford to replay or drop messages for downstream consumers.
  • SREs who want a cutover runbook that refuses to proceed on a half-caught-up mirror instead of leaving that judgment call to whoever is running the migration by hand.
  • Teams already using JetStream mirroring for disaster recovery who want the same mechanism to also carry live consumer cutovers during planned maintenance.

Why orchestrate this with Kestra

The nats CLI can read stream and consumer state and create consumers, but it has no concept of refusing an unsafe cutover, no cross-step memory of the sequences it captured, and no approval gate between "here is the plan" and "the target now has live consumers." Kestra supplies the safety gate as a first-class branch, the KV-backed audit record of exactly what was migrated and from where, the human approval pause, and a visible execution history proving the read-back verification actually happened.

Prerequisites

  • A source NATS cluster with JetStream enabled and the stream named in source_stream.
  • A target NATS cluster with a stream already configured to mirror the source (nats stream add <target_stream> --mirror=<source_stream>), created ahead of running this flow.
  • Network access from the Kestra worker to both clusters.
  • Durable (not ephemeral) consumers on the source stream, since ephemeral consumers have no stable ack_floor to migrate from once connections close.

Secrets

  • SOURCE_NATS_CREDS: NATS credentials file contents for the source cluster, passed as the NATS_CREDS environment variable the nats CLI reads natively. Leave unset if the source cluster has no auth configured.
  • TARGET_NATS_CREDS: NATS credentials file contents for the target cluster, same mechanism.
  • SLACK_WEBHOOK_URL: Slack incoming webhook used by every alert in this flow.
  • In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded, and read back in flows with {{ secret('NAME') }}.

Inputs

  • source_nats_url / target_nats_url (STRING, no default): NATS server URLs for each cluster.
  • source_stream / target_stream (STRING, no default): the source stream and its target mirror stream.
  • consumer_names (ARRAY of STRING, no default): the durable consumer names to migrate.

Outputs

  • outputs.migration_plan: the full captured per-consumer migration plan (consumer name, source ack-floor sequence) as JSON.

Quick start

  1. On the target cluster, create the mirror ahead of time: nats stream add {{target_stream}} --mirror={{source_stream}} and wait until nats stream info {{target_stream}} --json shows mirror.lag: 0.
  2. Add the SOURCE_NATS_CREDS / TARGET_NATS_CREDS (if auth is enabled) and SLACK_WEBHOOK_URL secrets.
  3. Run the flow manually with a small, non-critical consumer first and confirm evaluate_mirror_readiness reports is_safe: true.
  4. Resume pause_for_operator_approval and confirm verify_cutover reports every consumer verified: true with the expected starting sequence.
  5. Point the consuming application at the target cluster for that consumer, confirm it resumes cleanly with no gap or duplicate, then repeat for the remaining consumers.

How to extend

  • Add a cleanup task that calls nats consumer rm on the target for any consumer that failed verification, so a partial cutover never leaves a misconfigured consumer sitting there silently.
  • Extend evaluate_mirror_readiness to also refuse the cutover if the source stream is still receiving messages faster than some configured rate, mirroring the Kafka reference blueprint's "producer still writing" refusal check.
  • Loop this flow over multiple stream pairs for a full cluster migration instead of one stream at a time.

Pitfalls

  • This blueprint does not create the target mirror stream itself. The mirror must already exist and have caught up before this flow is run; evaluate_mirror_readiness only checks that state, it does not configure it.
  • Only durable consumers can be migrated this way. An ephemeral consumer's position lives only in the connection that created it and has no stable ack_floor to read once that connection is gone.
  • mirror.lag == 0 is a point-in-time check, not a guarantee the mirror stays caught up for the rest of the run. If the source stream keeps receiving new messages after inspect_source_and_mirror runs, the captured ack_floor sequences for each consumer are still correct (they describe what was already acknowledged, not the stream's end), but genuinely new messages arriving after cutover will only be visible once the mirror relays them to the target.
  • This flow never deletes or modifies anything on the source. There is no explicit rollback step because nothing on the source needs restoring; if verify_cutover fails, the fix is to remove the bad consumer on the target (see How to extend) and re-run, not to undo anything upstream.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.