MuleSoft API-Led Connectivity: SAGA Integration Pattern Demonstration Part 2

In the previous blog, I described two types of SAGA patterns: choreography and orchestration. I will use the following use case and requirements to illustrate how you can implement both types in MuleSoft.

Example: a company needs an e-commerce business solution where an order is placed in a transaction and a series of events occur that either a) result in the completion of the order or b) a failure occurs at some point and messages are raised that require compensating transactions to roll back the transaction effectively.

SAGA Choreography

I created three APIs and six Anypoint MQ queues to facilitate the implementation of choreography. The HTTP Listener employs TLS to secure the request in flight. The Anypoint MQ queues are set to use encryption at rest.

SAGA Choreography Flow Chart

demo-saga-orders-api

An order is initiated via Postman with an HTTPS Post to a listener endpoint /orders. The payload is enriched with status and orderId properties and values (“orderCreate” and “uuid()” respectively), then published to the order-queue. Since, by design, the processing from the order-queue is asynchronous, an immediate response is returned along with an HTTP status of 201 Accepted.

 

A separate flow, orders-main-subscriber, has an MQ Subscriber connector that listens on the orders-queue. It picks up the message and passes processing to orders-impl-process in the orders-impl.xml.

 

Based on the status of the payload, then it will be processed as follows:

1. payload.status equals “orderCreate”
   1. Transform the payload only to include orderId and status (other services can query additional details).
   2. Publish to the payment-queue
2. payload.status equals “Rollback”
   1. Transform the payload only to include orderId and status (other services can query additional details).
   2. Publish to the payment-queue
3. payload.status equals “orderComplete”
   1. Transform the payload to include orderId and status of “Order Completed”

demo-saga-payments-api

The flow, demo-saga-payments-api, has an MQ Subscriber connector that listens on the payments-queue. It picks up the message and passes processing to payments-impl-process sub-flow in payments-impl.xml.

Based on the status of the payload, it will be processed as follows:

 

1. payload.status equals “orderPayment”
   1. A flow reference is made to payments-impl-payment-success, and a variable paymentSuccess is set. This allows for testing both success and failure scenarios.
   2. A choice router is used to determine whether or not the payment was successful.
      1. vars.paymentSuccess is equal to true 
         1. Transform the payload to include orderId and set status to “orderInventory”
         2. Publish to the inventory-queue
      2. vars.paymentSuccess is equal to false
         1. Transform the payload to include orderId, set status to “Rollback” and set failureReason to “Payment Not Authorized”
         2. A flow reference calls the payments-impl-rollback sub-flow.
            1. Transform the payload to include orderId, set status to “Rollback” and set failureReason to “Payment Not Authorized”
            2. Publish to the orders-queue
2. payload.status equals “Rollback
   1. A flow-reference calls the payments-impl-rollback sub-flow and processing continues like 1.b.ii.2 above.

demo-saga-inventory-api

The flow, demo-saga-inventory-api, has an MQ Subscriber connector that listens on the inventory-queue. It picks up the message and passes processing to inventory-impl-process sub-flow in the inventory-impl.xml.

Based on the status of the payload, the message payload will be processed as follows:

1. payload.status equals “orderInventory”

1. 1. 1. A flow reference is made to inventory-impl-check-stock, and a variable inStock is set. This allows for testing both success and failure scenarios.
      2. A choice router is used to determine whether or not the inventory was successful.
         1. vars.inStock is equal to true
            1. Transform the payload to include orderId and set status to “orderComplete”
            2. Publish to the orders-queue
         2. vars.inStock is equal to false
            1. Transform the payload to include orderId, set status to “Rollback” and failureReason to “Inventory Out of Stock”
               1. Publish to the payments-queue

SAGA Orchestration

I created four APIs and eight Anypoint MQ queues to facilitate the implementation for orchestration. The HTTP Listener employs TLS to secure the request in flight. The Anypoint MQ queues and exchanges are set to use encryption at rest.

SAGA Orchestration Flow Chart

demo-saga-controller-api

An order is initiated via Postman with an HTTPS Post to a listener endpoint /orders. A flow reference to orders-impl is called.

The payload is enriched with a status property, and the value “orderCreate” is then published to the order-queue. Since, by design, the processing from the order-queue is asynchronous, an immediate response is returned along with an HTTP status of 201 Accepted.

A controller-impl.xml file contains the logic necessary to orchestrate the movement of events for each processing-related queue (order-queue, payment-queue, and inventory-queue).

The flow, controller-impl, has an MQ Subscriber connector that listens on the controller-queue. It picks up the message and evaluates the payload.status then takes appropriate action.

 

1. payload.status equals “orderSuccess”

1. 1. Transform the payload to include orderId and set status to “orderPayment”
   2. A flow reference calls the controller-impl-payment-queue sub-flow, which publishes the payload to the payment-queue
2. payload.status equals “paymentSuccess”
   1. Transform the payload to include orderId and set status to “orderInventory”
   2. A flow reference calls the controller-impl-inventory-queue sub-flow, which publishes the payload to the inventory-queue.
3. payload.status equals “inventorySuccess”
   1. Transform the payload to include orderId and set status to “orderComplete”
   2. A flow reference calls the controller-impl-order-queue sub-flow, which publishes the payload to the order-queue
4. payload.status equals “orderFailure”
   1. Transform the payload to include orderId, set status to “Rollback” and set reason to “Order Failure”
   2. A flow reference calls the controller-impl-exchange sub-flow, which publishes the payload to the order-exchange MQ Exchange (similar to a topic but where queues are subscribers).
   3. The order-exchange publishes the message to the three queues:  order-queue, payment-queue, and inventory-queue. By publishing the message to all three queues, it is up to the subscribers of those queues to decide to act on the message.
5. payload.status equals “paymentFailure”
   1. Payment failure is logged
   2. Transform the payload to include orderId, set status to “Rollback” and set reason to “Payment Failure”
   3. Same as 4b and 4c above.
6. payload.status equals “inventoryFailure”
   1. Inventory failure is logged
   2. Transform the payload to include orderId, set status to “Rollback” and set reason to “Inventory Failure”
   3. Same as 4b and 4c above.

demo-saga-orders-orch-api

The flow, demo-saga-orders-orch-api, has an MQ Subscriber connector that listens on the order-queue. It picks up the message and passes processing to orders-impl-process sub-flow in the orders-impl configuration.xml.

Based on the status of the payload, the message payload will be processed as follows:

1. payload.status equals “orderCreate”
   1. 1. A flow reference is made to orders-impl-order-success and a variable orderSuccess is set. This allows for testing both success and failure scenarios.
      2. A choice router is used to determine whether or not the inventory was successful.
         1. vars.orderSuccess equals true
            1. Transform the payload to include orderId and set status to “orderSuccess”
            2. Publish to the controller-queue
         2. vars.orderSuccess equals false
            1. Transform the payload to include orderId, set status to “Rollback” and set reason to “Order Failure”
               1. Publish to the controller-queue queue.
2. payload.status equals “Rollback”
   1. 1. Some processing occurs to “roll back” the order.

demo-saga-payments-orch-api

The flow, demo-saga-payments-orch-api, has an MQ Subscriber connector that listens on the payments-queue. It picks up the message and passes processing to payments-impl-process sub-flow in the payments-impl configuration.xml.

Based on the status of the payload, the message payload will be processed as follows:

 

1. payload.status equals “orderPayment”
   1. A flow reference is made to payments-impl-payment-success sub-flow and a variable paymentSuccess is set. This allows for testing both success and failure scenarios.
   2. A choice router is used to determine whether or not the inventory was successful.
      1. vars.paymentSuccess equals true
         1. Transform the payload to include orderId and set status to “paymentSuccess”
         2. Publish to the controller-queue.
      2. vars.paymentSuccess equals false
         1. Transform the payload to include orderId, set status to “paymentFailure” and set failureReason to “Payment Not Authorized”
         2. Publish to the controller-queue queue
2. payload.status equals “Rollback”
   1. Some processing occurs to “roll back” the payment.

demo-saga-inventory-orch-api

The flow, demo-saga-inventory-orch-api, has an MQ Subscriber connector that listens on the inventory-queue. It picks up the message and passes processing to inventory-impl-process sub-flow in the inventory-impl configuration.xml.

Based on the status of the payload, the message payload will be processed as follows:

1. payload.status equals “orderPayment”
1. A flow reference is made to inventory-impl-payment-success sub-flow and a variable inStock is set. This allows for testing both success and failure scenarios.
2. A choice router is used to determine whether or not the inventory was successful.
1. vars.inStock equals true
1. Transform the payload to include orderId and set status to “inventorySuccess”
2. Publish to the controller-queue
2. vars.inStock equals false
1. Transform the payload to include orderId, set status to “paymentFailure” and set failureReason to “Inventory Out Of Stock”
2. Publish to the controller-queue queue.

Summary

As you can see, both Choreography and Orchestration are viable enterprise integration patterns when you need to ensure data consistency in a distributed system without tight coupling and be able to rollback or compensate if one of the operations in the sequence fails.

Choreography lends itself to a smaller number of services whereas Orchestration is more appropriate for many services and scales much better when adding new services.

Depending on the use case, you will need to decide which SAGA method is most appropriate for you.