Netflix Conductor: Inversion of Control for workflows
In December 2016, we open sourced Netflix Conductor.
We have been working hard since then to add more features, enhance the user interface for monitoring the workflows, and harden the system. We have seen an increase in usage, new use cases, feature requests, and lot of interest from the community. In this post, we will talk about a couple of major features we added to Conductor this quarter.
IoC in Conductor
Conductor makes it possible to reuse workflows and embed them as sub workflows. Sub workflows are a great way to promote process reuse and modularity. As Netflix Conductor adoption increased, we observed interesting use cases where business processes were initiated based off the events and state changes in other workflows.
Traditionally, applications implemented such use cases using pub/sub systems and publishing events as application state changed and subscribing to interested events to initiate actions.
We sought a solution with the following characteristics:
- Loose coupling between workflows
- Allow starting a workflow based on state changes in other workflows
- Provide integration with external systems producing / consuming events via SQS/SNS
Enter Inversion of Control — for workflows. The idea is to be able to chain workflow actions such as starting a workflow, completing a task in a workflow, etc. based on state changes in other workflows.
To illustrate this further, let’s take a look at two solutions to trigger a “QC Workflow” after a file has been ingested. Once a file has been ingested, multiple workflows are triggered (owned by separate applications); one of them is a file verification workflow (QC Workflow) that is triggered to identify the type of the file (Audio, Video etc.), run appropriate checks, and start the encoding process for the file.
With Conductor, there are two separate ways to achieve this; both the solutions get the job done, but there are subtle differences in both the approaches.
Starting another workflow as sub workflow
Given that sub workflows are tasks, their output can be consumed by other tasks in the File Ingest workflow subsequent to the QC workflow, e.g. outcome of the QC process. Sub workflows thus are useful when there is a tight dependency and coupling between business processes.
Starting a workflow on an event
In the above approach, the File Ingest workflow produces an “Ingest Complete” event. This event can be consumed by one or more event handlers to execute actions — including starting of a workflow.
- File Ingest workflow does not have any direct workflow dependencies
- Workflows can be triggered based on events promoting loose coupling
- The outcome of QC Task workflow does not impact the File Ingest workflow
- Multiple workflows or actions can be executed per each event
EVENT Task
We introduced a new type of task called EVENT. An event task can be added to the workflow definition. When executed, it produces an event in the specified “sink”. A sink is an eventing system such as SQS, Conductor, or any other supported systems. Sinks follow a pluggable architecture where systems such as JMS, Kafka etc. can be added by implementing the required interfaces and making the implementation available in the classpath of the Conductor server JVM.
Following the task input / output model of conductor, the input to the EVENT task is calculated and is sent as payload. The sink can be either Conductor or an external system such as SQS. The architecture for supporting various types of Sink is plugin based and new types such as JMS or Kafka can be easily added by implementing the required interfaces and making it available in the JVM classpath for Conductor server.
Below is an example of an event task that publishes an event to an SQS queue identified by name.
{
"name": "example_event",
"taskReferenceName": "event0",
"input": {
"filename": "${workflow.input.filename}",
"location": "${anothertask.output.location}"
},
"type": "EVENT",
"sink": "sqs:sqs_queue_name"
}Event Handlers
Event handlers are listeners that are executed upon arrival of a specific event. Handlers listen to various event sources including Conductor and SQS/SNS and allows users to plug in other sources via APIs. Conductor supports multiple event handlers per event type. Event handlers are managed with Conductor using the /event API endpoints. It is possible to have multiple event handlers for a single event source.
Event Condition
Condition is a JavaScript expression on the payload, which MUST evaluate to true for the event handler to execute the actions. The condition acts as a the filter to selectively take actions on a subset of events matching the criteria. Conditional filters are optional and when not specified all the events from the source are processed.
Event Actions
Each Event handler has one or more actions associated with it. When the condition associated with an event is evaluated to “true”, the actions are executed. The supported actions are:
- Start a workflow
- Complete a task as failed or success
Example
Below a sample event handler that listens to an SQS queue and starts the workflow based on the condition
{
"name": "conductor_event_test",
"event": "sqs:example_sqs_queue_name",
"condition": "$.file_type == ‘image’",
"actions": [
{
"action": "start_workflow",
"start_workflow": {
"name": "workflow_name",
"input": {
"file_name": "${filename}",
"file_type": "${file_type}"
}
}
}
],
"active": true
}Conductor UI
All the registered event handlers can be inspected in the UI. Updates can be made via REST APIs or through the swagger API page.
JSON Data Transformation
Conductor now supports JSONPath based data transformation. Using JSON Path in the input configuration to task allows complex data transformation and reduces the need to write one-off tasks that otherwise does the data transformation.
https://netflix.github.io/conductor/metadata/#wiring-inputs-and-outputs
Path forward
While it’s been a busy quarter, we are not done yet. Looking forward to Q2, our focus is going to be on making it easier to test the workflows for the developers and support for logging task execution context to help understand and troubleshooting workflows spanning multiple workers and applications.
If you like the challenges of building distributed systems and are interested in building the Netflix studio ecosystem and the content pipeline at scale, check out our job openings.
