Event-Driven Microservices With Orkes Conductor

1. Introduction

In this tutorial, we’ll explore how to build event-driven microservices using the Orkes Conductor and Spring. We’ll use Conductor to orchestrate microservices using HTTP endpoints and service workers.

2. Event-Driven Microservices

Microservices offer a great way to create a modular architecture that can be scaled and managed independently. Developers typically design microservices as single-responsibility services that perform one thing exceptionally well. However, an application flow typically requires coordination across multiple microservices to achieve the business goals.

Event-driven architecture robustly facilitates communication among microservices over eventing systems, ensuring scalability and durability of the flows. For these reasons, event-driven microservices have recently gained popularity and are especially useful when implementing asynchronous flows.

2.1. Shortcomings of Event-Driven Systems

While great at decoupling service interaction, with event-driven systems come several challenges:

• Difficult to visualize the execution flow – All the communication among microservices happens over the event bus, so it’s difficult to visualize and reason about the business flows. This makes it harder to identify, debug, and recover from failures. Often, distributed traces and centralized logging are used to solve the problem.
• No single authority for the application state – Typically, each service maintains its local database, which acts as a system of record for that service. For example, a credit card service could have a database with a list of credit card payments and relevant information. However, across multiple service calls, the overall state of the application is distributed, making it difficult to visualize application flow, handle compensating transactions in case of failures, and query the application’s state at a given time.
• Easy to build, difficult to scale – Frameworks like Spring simplify the building of event-driven applications that can connect to various pub/sub systems. However, developers often invest a significant amount of time addressing challenges such as operationalizing the systems, scaling to handle large workloads, or building applications with very complex connectivity rules.

3. Event-Driven Architecture With Conductor

Netflix originally built Conductor as a platform for orchestrating microservices. Developers at Netflix designed and built Conductor to create event-driven microservices and to address some of the shortcomings listed above.

As an orchestrator, Conductor allows us to define the flow of service executions either in code or in JSON and enables us to wire up services or write service workers using any of the supported language SDK. Conductor, as a fully open-source platform, operates under the Apache 2.0 license.

Conductor’s polyglot nature allows us to write service workers in any language or have services and workers in different languages, even within a single application flow.

Conductor lets us create re-usable, single responsibility principle event-driven services that respond to events. Conductor can also be used to wire up services that are exposed over HTTP using persisted queues.

4. Event-Driven Microservices With Conductor and Spring

Now, let’s explore an example Spring Boot application that leverages Conductor to orchestrate across microservices.

4.1. Setting up Orkes Conductor

Orkes Conductor can be configured in various ways. First, we can set it up locally using Docker, or alternatively, we can utilize the free developer sandbox Playground.

There’s also a Slack community available that might be a good place to check out with any queries related to Conductor.

4.2. Method 1 – Installing and Running Locally Using Docker

First, let’s ensure that Docker is installed on the device.

Then, we employ the following Docker command to initiate the server on port 9090 and the UI on port 1234:

docker run --init -p 9090:8080 -p 1234:5000 --mount source=redis,target=/redis \
--mount source=postgres,target=/pgdata orkesio/orkes-conductor-community-standalone:latest

Let’s create a simple Spring Boot application that does two things:

• Create a Microservice worker using Conductor.
• Orchestrate between these two services:
  • An HTTP endpoint https://orkes-api-tester.orkesconductor.com/api
  • The service worker we created in the first step.

Here’s how we can create a simple service worker using a task worker in Conductor that doesn’t need to be exposed over an HTTP endpoint:

@WorkerTask(value = "fraud-check-required")
public FraudCheckResult isFraudCheckRequired(BigDecimal amount) {
    return fraudCheckService.checkForFraud(amount);
}

Let’s create a simple workflow that calls a sample HTTP endpoint to get customer details (https://orkes-api-tester.orkesconductor.com/api) and the service worker that runs the fraud check we just implemented above in parallel.

We execute the workflow using the following command, resulting in a workflow accessible at http://localhost:1234/workflowDef/microservice_orchestration:

curl -X 'POST' 'http://localhost:9090/api/metadata/workflow' \
     -H 'accept: */*' \
     -H 'Content-Type: application/json' \
     -d '{
         "name": "microservice_orchestration",
         "description": "microservice_orchestration_example_workflow",
         "version": 1,
         "tasks": [
             {
                 "name": "fork_task",
                 "taskReferenceName": "fork_task_ref",
                 "inputParameters": {},
                 "type": "FORK_JOIN",
                 "forkTasks": [
                     [
                         {
                             "name": "fraud-check-required",
                             "taskReferenceName": "fraud-check-required_ref",
                             "inputParameters": {
                                 "amount": "${workflow.input.amount}"
                             },
                             "type": "SIMPLE"
                         }
                     ],
                     [
                         {
                             "name": "get_customer_details",
                             "taskReferenceName": "get_customer_details_ref",
                             "inputParameters": {
                                 "http_request": {
                                     "uri": "https://orkes-api-tester.orkesconductor.com/api",
                                     "method": "GET",
                                     "accept": "application/json",
                                     "contentType": "application/json"
                                 }
                             },
                             "type": "HTTP"
                         }
                     ]
                 ]
             },
             {
                 "name": "join_task",
                 "taskReferenceName": "join_task_ref",
                 "type": "JOIN",
                 "joinOn": [
                     "get_customer_details_ref",
                     "fraud-check-required_ref"
                 ]
             }
         ],
         "inputParameters": [
             "amount"
         ],
         "schemaVersion": 2,
         "restartable": true
     }'

Let’s run the newly created workflow by making an HTTP POST request:

curl -X 'POST' \
'http://localhost:9090/api/workflow/microservice_orchestration' \
-H 'accept: text/plain' \
-H 'Content-Type: application/json' \
-d '{
    "amount": 1000.00
}'

We can verify the completed execution by navigating to “Executions” on the Orkes Conductor UI and checking the workflow execution ID.

Now, let’s delve into how we can employ this orchestration across services in our application. We’ll expose an endpoint that executes this workflow, effectively creating a new API endpoint that orchestrates the microservices using event-driven design.

Here’s the sample command:

curl -X 'POST' \
'http://localhost:8081/checkForFraud' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
    "accountId": "string",
    "amount": 12
}'

4.3. Method 2 – Using Orkes Playground

Let’s create a free account and leverage Playground to test Conductor in real-time by following these steps:

1. Login to https://play.orkes.io/.
2. Create an account to get started with Orkes Conductor.

Now, let’s create a new workflow in Playground or, for ease of testing, we can also use the following workflow:

View Workflow in Playground

In order to create a connection between Orkes Playground and the worker, we need to create an application in Orkes Conductor. Let’s follow these steps:

1. On Orkes Playground, navigate to Access Control > Applications.
2. Click ‘Create Application‘ and provide an app name.
3. Choose the ‘Application role‘ as ‘Worker‘.
4. Click ‘Create access key‘ and copy and keep the key ID & key secret.

Next, let’s grant access to run the workflow by following these steps:

1. Under the ‘Permissions‘ section, we click ‘+Add permission‘.
2. Under the ‘Workflows‘ tab, we choose ‘microservice_orchestration‘, and under the ‘Tasks‘ tab, let’s choose ‘fraud_check_required‘
3. Choose ‘EXECUTE‘ permission and add permissions.

Now, let’s open the worker, and under the application.properties file, provide the generated key ID & secret. We should replace conductor.server.url with https://play.orkes.io/api:

conductor.security.client.key-id=your_key_id
conductor.security.client.secret=your_key_secret
conductor.server.url=https://play.orkes.io/api

Let’s run the application. We can see that the worker polls for the Conductor tasks and picks the task once available.

Now, we use the http://localhost:8081/checkForFraud endpoint that we created in our Spring Boot application, and it will use play.orkes.io as the Conductor backend server to run the workflow.

5. Conclusion

Event-driven microservices open up exciting possibilities for building scalable and responsive software systems. In this article, we’ve gone through the fundamentals of event-driven microservices, highlighting their advantages and challenges.

We’ve explored how microservices, with their modular and single-responsibility nature, offer an excellent foundation for creating complex applications.

As always, the source code for the article is available over on GitHub.