MuleSoft API-Led Connectivity: SAGA Integration Pattern

Enterprise integration patterns play a significant role in event-driven development and are accepted solutions in many contexts. These patterns fit well into MuleSoft’s API-led connectivity architecture by providing a framework for designing and building messaging and integration systems across distributed services. Additionally, MuleSoft reduces the effort required when building integrations by supporting enterprise integration patterns like SAGA.

What is a SAGA pattern?

A SAGA pattern is a sequence of local transactions where each transaction updates data within a single service. The first transaction in a SAGA is initiated by an external request corresponding to the system operation. Then each subsequent step is triggered by the completion of the previous one.

Why use a SAGA pattern?

In some use cases, transactions must be atomic, consistent, isolated, and durable (ACID). Transactions within a single service can be ACID (using transactional strategies like 2-Phase commit), but transactions that span multiple services require a different transaction management strategy. The SAGA pattern is a way to achieve reliability that does not have ACID guarantees.

SAGA provides transaction management using a sequence of local transactions. A local transaction is a single work effort performed by a SAGA participant (service). Each local transaction within a service performs a business or system function and publishes a message or event to trigger the next service’s local transaction in the SAGA. If a local transaction fails, the SAGA executes a series of compensating transactions to undo the changes that were made by the preceding services.

The following are two ways of implementing SAGA:

• Choreography: When there is no coordinator control, each service produces and listens to other service’s events and decides if an action should or should not be taken.
• Orchestration: When a coordinator service is responsible for centralizing the SAGA’s routing of messages/events to appropriate services.

Use Case

Let’s examine each of these with a simple use case. Consider an e-commerce example where an order is placed in a transaction and a series of events occur that either a) result in the completion of the order or b) a failure occurs which raises messages that require compensating transactions to roll back the transaction.

Requirements

1. Functional
   1. The original message should be published into a Dead Letter Queue for manual inspection.
   2. The message status and reason need to be communicated to any upstream services, and any transactional activity needs to be rolled back.
   3. An order transaction must be communicated to the other downstream services synchronously, maintaining the Order ID and Status throughout the life of the transaction.
   4. Orders do not need to be processed in the order in which they are received.
   5. If an error occurs with any of the services:
   6. Error messages can be delivered to upstream services either synchronously or asynchronously.
   7. Successful preparation and shipment of the order must be communicated to the Order Service.
2. Non-Functional
   1. All messages must be secure in transit and at rest.
   2. Messages must be held for delivery for one week then moved to a default DLQ.
   3. Messages cannot be lost.
   4. Processing must be highly available.
   5. Transactions eventually must be consistent.

Because we are looking at inter-service event communication, it is important to use an appropriate messaging service. A message broker service is a great solution when non-functional requirements dictate reliability, availability, and scalability.

A few popular message broker services include Anypoint MQ, ActiveMQ, and RabbitMQ. MuleSoft provides connectors for these with the Anypoint MQ connector and the JMS connector.

Based on the requirements for this use case, we can analyze what features we need.

Message Broker Feature	Must-Have	Nice to Have
FIFO Queues		X
Topics		X
Message TTL	X
Default Ack Timeout		X
Default Delivery Delay		X
Encryption At Rest	X
Default Dead Letter Queue	X
High Availability	X
Persistence	X
MuleSoft Connector Available	X

 

Next, we’ll examine the different SAGA approaches.

Choreography

Using the Choreography approach:

The Order Service receives a new order and publishes an Order Success message. The Order Service listens to Payment Failure, which, if received, handles it by creating a compensating transaction (rollback). It also listens to Inventory Success and finishes the order by setting the status to Complete.

The Payment Service listens to Order Success messages. It processes a payment and publishes a Payment Success message. If the payment fails, it publishes a Payment Failure message. The Payment Service listens to Inventory Failure, which, if received, handles it by creating a compensating transaction then publishes a Payment Failure message.

The Inventory Service listens to Payment Success messages. It prepares the order for shipment and publishes an Inventory Success message. If the inventory preparation fails (e.g., out of stock) it publishes an Inventory Failure message.

Benefits

• Good for simple workflows that require few participants and don’t need coordination logic
• Doesn’t require additional service implementation and maintenance.
• Doesn’t introduce a single point of failure since the responsibilities are distributed across the SAGA participants
• Inherits benefits of event-based solutions such as Asynchronous Communication and Loose Coupling
• Eventual consistency

Drawbacks

• Workflow can become confusing when adding new steps, as it’s challenging to track which SAGA participants listen to which commands
• There’s a risk of cyclic dependency between SAGA participants because they consume each other’s messages
• Integration testing is difficult because all services must be running to simulate a transaction

Orchestration

Using the Orchestration approach:

The Orchestration Service receives an order transaction.

The Order Service listens for messages from the Orchestrator. For order messages, it creates the order and publishes a success message to the Orchestrator. If the order creation fails, it publishes a failure message to the Orchestrator. The Orchestrator publishes a rollback message to the Order Service, which creates a compensating transaction if applicable. If it receives an Inventory Success message from the Orchestrator it finishes the order by setting the order status to Complete.

The Payment Service listens for payment messages from the Orchestrator. It processes a payment and publishes a success message to the Orchestrator. If the payment fails, it publishes a payment failure message to the Orchestrator, which in turn publishes a rollback message for the Order Service to create a compensating transaction.

The Inventory Service listens for inventory messages from the Orchestrator. It prepares the order for shipment and publishes a success message for the Orchestrator. If the inventory preparation fails (e.g., out of stock) it publishes an inventory failure message for the Orchestrator, which publishes a rollback message for the Payment Service and the Order Service to create compensating transactions.

Benefits

• Good for complex workflows involving many participants or new participants added over time
• Suitable when there is control over every participant in the process and control over the flow of activities
• Doesn’t introduce cyclical dependencies because the Orchestrator unilaterally depends on the SAGA participants
• SAGA participants don’t need to know about commands for other participants. Clear separation of concerns simplifies business logic
• Eventual consistency

Drawbacks

• Additional design complexity requires the implementation of coordination logic
• There’s a single point of failure because the Orchestrator manages the complete workflow
• Can be more expensive. An additional worker(s) is required for the Orchestration service

Summary

SAGA is a great integration pattern for transactional logic that spans multiple services and where consistency is desired. Using a message broker like Anypoint MQ, you can also achieve security, reliability, availability, and scalability.

Use the SAGA pattern when you need to:

• Ensure data consistency in a distributed system without tight coupling
• Rollback or compensate if one of the operations in the sequence fails

Two types of SAGA patterns are:

• Choreography

• Orchestration

The SAGA pattern is less suitable for:

• Tightly coupled transactions
• Compensating transactions that occur in earlier participants
• Cyclic dependencies

In Part 2 of this blog, we’ll look at a way to implement both the Choreography and Orchestration patterns for this use case using MuleSoft and Anypoint MQ.