Azure AI Foundry Evaluation Failure Gate

An AI Foundry evaluation run that fails or gets cancelled tells nobody on its own; the studio shows the outcome if someone opens the run, and Azure AI Foundry does not keep a schedule, decide which statuses count as a problem, or notify anyone the moment one comes back anything but Completed. This blueprint watches every newly terminal evaluation and pages the moment one does not succeed, with the real evaluation name attached.

This blueprint was created by Mohit Jeswani.

How it works

1. watch_evaluations (io.kestra.plugin.azure.aifoundry.Trigger) polls Azure AI Foundry's evaluations API every 5 minutes and fires one execution per batch of evaluations that newly reached Completed, Failed, Canceled, or Expired, deduplicating already-observed evaluations via stateful tracking.
2. evaluate (io.kestra.plugin.core.output.OutputValues) counts how many of this batch's evaluations did not complete successfully and classifies the result: OK when all succeeded, EVALUATION_FAILED for one or a few, WIDESPREAD_EVALUATION_FAILURE at or above max_failed_in_batch.
3. route (io.kestra.plugin.core.flow.If):
   • compliant: logs that every evaluation in the batch succeeded.
   • alert: posts the failed count and the real evaluation names with their statuses to Slack.
4. summary logs the final counts and severity, closing the governance cycle.
5. errors alerts Slack if the flow itself fails, for example an expired service principal secret.

What you get

• A failure check built on the evaluation's own terminal status, not a dashboard someone has to remember to check after kicking off a run.
• The real evaluation names and statuses in the Slack alert, not just a count.
• A stateful trigger that never re-fires for the same evaluation twice, so one failure produces one alert, not a repeated page on every poll.

Who it's for

• AI platform and ML ops teams running evaluation pipelines on Azure AI Foundry who want a failed or cancelled run paged the moment it happens, not discovered when someone checks the studio days later.
• Teams evaluating the Azure plugin who want a worked example of aifoundry.Trigger used as a standing evaluation-failure gate, distinct from this repo's existing Azure AI Foundry blueprint, which audits a deployment's model configuration rather than reacting to evaluation run outcomes.

Why orchestrate this with Kestra

Azure AI Foundry reports an evaluation's status on request; it does not decide which statuses mean trouble, compare a batch of failures against a tolerance you declare, or notify a Slack channel the moment one appears. Kestra keeps max_failed_in_batch as a flow input, turns a failed evaluation into a Slack page naming the real run, and keeps a full execution history of every terminal evaluation ever observed.

Prerequisites

• An Azure AI Foundry project that runs model or agent evaluations.
• An Azure Entra ID service principal with read access to that project's evaluations.
• A Slack incoming webhook URL.

This flow calls the Azure AI Foundry API directly over HTTPS; there is no official local Azure AI Foundry emulator, so watch_evaluations cannot be exercised against a local Docker container.

Secrets

• AZURE_AI_FOUNDRY_ENDPOINT: the project endpoint URL. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_AZURE_AI_FOUNDRY_ENDPOINT.
• AZURE_TENANT_ID / AZURE_CLIENT_ID / AZURE_CLIENT_SECRET: the Entra ID service principal credentials used by watch_evaluations. In Kestra OSS, provide them as the base64-encoded environment variables SECRET_AZURE_TENANT_ID, SECRET_AZURE_CLIENT_ID, and SECRET_AZURE_CLIENT_SECRET.
• SLACK_WEBHOOK_URL: used by alert and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Default	Purpose
max_failed_in_batch	3	Failed-evaluation count per poll at or above which the batch is classified as widespread

Outputs

• {{ trigger.evaluation }} / .evaluations / .total: the newly terminal evaluations this poll found (each with name, status), as the Trigger reports them.
• {{ outputs.evaluate.values.failed_count }} / .failed_names / .total_evaluations: how many evaluations failed, their names with statuses, and the total observed this poll.
• {{ outputs.evaluate.values.severity }}: OK, EVALUATION_FAILED, or WIDESPREAD_EVALUATION_FAILURE.

Pitfalls

• statuses on the trigger defaults to Completed, Failed, Canceled, and Expired; an evaluation stuck in a non-terminal running state indefinitely never fires this trigger at all, since it is not one of those four states.
• The trigger's state is deduplicated by evaluation name and status; an evaluation that is somehow re-run under the exact same name and reaches the same status again may not re-fire depending on on mode and stateTtl.
• This trigger has no inputs: to override per execution; endpoint, credentials, and interval are trigger-level configuration set directly on the trigger block.

Quick start

1. Add the four Azure secrets and SLACK_WEBHOOK_URL.
2. Set endpoint on watch_evaluations to a real AI Foundry project.
3. Enable the trigger, run a real evaluation, and confirm compliant fires once it completes successfully.
4. Run (or cancel) an evaluation you expect to fail to confirm alert fires as EVALUATION_FAILED.

How to extend

• Chain an AI Foundry evaluation-results read task on each failed evaluation name to include the actual failure reason in the Slack alert, not just the status.
• Route WIDESPREAD_EVALUATION_FAILURE to an on-call paging channel and the isolated case to a lower-urgency channel, since a batch failure usually points at an infrastructure issue rather than one bad test case.
• Lower interval for evaluation pipelines with a tight feedback-loop requirement, or raise it for infrequent, long-running evaluation jobs.

Links

• Azure plugin: https://kestra.io/plugins/plugin-azure
• Azure AI Foundry evaluations reference: https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-approach-gen-ai
• Slack incoming webhooks: https://api.slack.com/messaging/webhooks