​Build a ​Custom ​Plugin for ​Kestra

Build your own Custom Plugin for Kestra.

In this tutorial, we will be building a custom plugin in Kestra.

Use-case for Custom Plugin

We will be building a plugin that fetches the data for a given pokemon. We will use the API provided by PokeAPI.co to fetch the pokemon's details: https://pokeapi.co/api/v2/pokemon/{pokemon_name}.

The API provides detailed information about any pokemon. We will fetch a few fields like the ability names, base experience, height and move names, and showcase it in the output of our plugin. The plugin task will accept the pokemon name, and return the selected fields in the output. This is how the task should look:

id: fetch_details
type: io.kestra.plugin.pokemon.Fetch
pokemon: pikachu

Requirements

You will need the following installed on your machine before proceeding:

• Java 21 or later.
• IntelliJ IDEA (or any other Java IDE, we provide only help for IntelliJ IDEA).
• Gradle (included most of the time with the IDE).

Create a new plugin

Here are the steps:

1. Go on the plugin-template repository.
2. Click on Use this template.
3. Choose the GitHub account you want to link and the repository name for the new plugin.
4. Clone the new repository: git clone [email protected]:{{user}}/{{name}}.git.
5. Open the cloned directory in IntelliJ IDEA.
6. Enable annotations processors.
7. If you are using an IntelliJ IDEA < 2020.03, install the lombok plugins (if not, it's included by default).

Once you completed the steps above, you should see a similar directory structure:

As you can see, there is one generated plugin: the Example class representing the Example plugin (a task).

A project typically hosts multiple plugins. We call a project a group of plugins, and you can have multiple sub-groups inside a project by splitting plugins into different packages. Each package that has a plugin class is a sub-group of plugins.

Gradle Configuration

We use Gradle as a build tool.

Mandatory configuration

The first thing we need to configure is the plugin name and the class package.

1. Change in settings.gradle the rootProject.name = 'plugin-template' with the plugin name rootProject.name = 'plugin-pokemon'.
2. Change the class package: by default, the template provides a package io.kestra.plugin.templates, just rename the templates folder in src/main/java & src/test/java to pokemon. And change the first line in Example.java, ExampleRunnerTest.java and ExampleTest.java to package io.kestra.plugin.pokemon;.
3. In build.gradle: a. replace description 'Plugin template for Kestra' to the package name description 'Plugin pokemon for Kestra'. b. In the dependencies section, add a dependency which we will be using in our plugin task: implementation group: 'com.googlecode.json-simple', name: 'json-simple', version: '1.1.1' c. Change the jar section to the following:

jar {
    manifest {
        attributes(
                "X-Kestra-Name": project.name,
                "X-Kestra-Title": "Pokemon",
                "X-Kestra-Group": project.group + ".pokemon",
                "X-Kestra-Description": project.description,
                "X-Kestra-Version": project.version
        )
    }
}

Develop Fetch Task

Create Pokemon class

In src/main/java/io/kestra/plugin/pokemon, we will create a new class Pokemon.java. This will be used to map the JSON output of the pokemon API to the Java class. We only need to add the fields that we are interested in, and ignore the rest.

Runnable Task

We will refactor the java file from Example.java to Fetch.java.

In this file, we will put in the appropriate schema for the plugin, including the inputs and output of the plugin. This will help us generate documentation for the plugin too. Also, we will include a few examples to help users understand how to use the plugin.

The class should extend Task and implement RunnableTask for it to be considered as a plugin task. The RunnableTask has a generic representing the output class. The output class should implement io.kestra.core.models.tasks.Output.

The actual crux of the task logic resides in the run method. This is an override method from the Task class, and takes RunContext as an argument, and return the instance of the Output class.

In the run method, we use the name of the pokemon, and make a call to the pokemon API. The fetched response is then mapped to the Pokemon class using the ObjectMapper. The resulting Pokemon object is then transformed into the Fetch.Output class, and returned.

Compile the plugin

Now that the plugin is developed, let us see how we can package it and test it on Kestra instance. Use the included Gradle task to build the plugin.

To build your plugin, execute the ./gradlew shadowJar command from the plugin directory.

The resulting JAR file will be generated in the build/libs directory.

To use this plugin in your Kestra instance, add this JAR to the Kestra plugins path.

Writing unit tests

Let us refactor the file names from ExampleRunnerTest.java and ExampleTest.java to FetchRunnerTest.java and FetchTest.java respectively. Under the tests/resources/flows folder, we have a file example.yaml, which we will rename to pokemonFetch.yaml.

Let us use a relevant flow in pokemonFetch.yaml.

Let us now amend the FetchRunnerTest.java. In this test, we try to load the flow file pokemonFetch.yaml, and run this flow. We then test if all the tasks in this flow were executed.

Let’s test the actual logic of the plugin in FetchTest.java. Here, we are creating the input and invoking the logic present in the task, and then verifying the output returned by the task.

Running the tests

You can run the tests from Intellij IDE, or from the terminal using the command:

./gradlew test

Plugin in Action

Now that the plugin is developed and tested, its time to see the plugin in action.

Use a custom docker image with your plugin

Add this Dockerfile to the root of your plugin project:

FROM kestra/kestra:develop

COPY build/libs/* /app/plugins/

You can build and run the image with the following command assuming you're in the root directory of your plugin: ./gradlew shadowJar && docker build -t kestra-custom . && docker run --rm -p 8080:8080 kestra-custom server local

You can now navigate to http://localhost:8080 and start using your custom plugin.

Execute the plugin and check the Output

Create a new flow, and use this newly-built plugin's task in the flow. Here is a sample flow:

id: pokemonFetch
namespace: company.team

tasks:
- id: fetch-pikachu
  type: io.kestra.plugin.pokemon.Fetch
  pokemon: "pikachu"

On executing the flow, you can go to the Outputs tab, and check out the output.

You are now all set to build more plugins and explore Kestra to its fullest!