Electronic Data Interchange (EDI) formats are commonly used in many industries. They have been in use for a long time and have well-defined data structures to transfer business-to-business (B2B) transaction data across systems. In earlier B2B posts, we discussed how using the Mulesoft Anypoint Partner Manager Can Accelerate Your Business.
In this blog, I will explore an approach for transforming an 856 Advanced Shipment notice document into a JSON format and vice-versa.
X12 and other EDI formats define document types such as 850 Purchase Order, 836 Procurement Notices, and many more. This list of predefined document types include industry-specific documents such as HIPAA documents for the healthcare industry. In our previous blog, How to Do B2B Integrations Using MuleSoft EDI X12 Module, we demonstrated how to transform X12 810 Invoice documents into JSON for processing within your APIs.
When working with X12 documents in MuleSoft, we leverage Mule’s X12 Connector that allows us to read and write documents in and out of X12 format. The connector consumes a standard X12 text-based document and converts it into a DataWeave-friendly data structure for transforming it into other API-friendly formats such as JSON or XML.
Some documents such as 850 Purchase Order are easy to map into an API-friendly format, but documents like 856 Advanced Shipment notices can be tricky due to the hierarchical data presented in a plain-text format. Although the X12 connector can convert it into DataWeave friendly structure, the challenge of transforming this hierarchical data into an API-friendly structure may require some advanced DataWeave usage.
X12 856 Advanced Shipment Notice
Let’s consider an example of an 856 document that contains Shipment notices and details which are presented in a hierarchy. The Hierarchical Level (HL) segments describe four types of data elements: Shipment, Orders, Packages, and Items.
ST - 856
(1-to-N) → HL - Shipment
(1-to-N) → HL - Order
(1-to-N) → HL - Package
(1-to-N)→ HL - ItemHL Segment has three elements to define the hierarchy:
– HL01: Defines the current level in the hierarchy
– HL02: Defines the parent hierarchy level
– HL03: Type of the hierarchy item – Shipment (S), Order (O), Package (P), Item (I).
ISA|00| |00| |ZZ|123456X |ZZ|9999999
|201229|1539|U|00400|012291501|0|P|<
GS|SH|123456X|9999999|20201229|1539|1399|X|004010
ST|856|000000001
BSN|00|020201229153948475|20201229|153948|0001
HL|1||S
TD1|CTN|1||||G|19.72|LB
TD5||2|UPSN
REF|BX|11942132941
DTM|011|20201229|0839|GM
DTM|017|20201231|0839|GM
FOB|CC
N1|SF|A. VENDOR INC.|15|123456X
N3|12345 West 100th Street
N4|BOCA RATON|FL|12345|US
N1|ST|A. CUSTOMER|15|99999861
N3|123 ANY ROAD|WAREHOUSE
N4|NEW YORK|NY|12345|USA
HL|2|1|O
PRF|4NY1IIZE
HL|3|2|P
REF|CN|R75E580A0301138041
MAN|GM|00008827693741579888
HL|4|3|I
LIN|0|EN|9780815345558|||PO|123ZZ456
SN1||1|EA
HL|5|1|O
PRF|123ZZ456
HL|6|5|P
REF|CN|R75E580A0301138050
MAN|GM|00008827693741579895
HL|7|6|I
LIN|0|EN|9780415718981|||PO|123ZZ456
SN1||4|EA
HL|8|1|O
PRF|123ZZ456
HL|9|8|P
REF|CN|R75E580A0301138069
MAN|GM|00008827693741579918
HL|10|9|I
LIN|0|EN|9781596671997|||PO|123ZZ456
SN1||16|EA
CTT|10|16
SE|41|000000001
GE|1|1399
IEA|1|012291501For the purpose of this use case, let’s assume that our backend API accepts Shipment data in the JSON format. The JSON structure contains a Shipment object with the hierarchy of Shipment → Orders → Packages → Items.
{
"shipment": {
//Shipment Info goes here
},
"orders": [
//Collection of all orders in this Shipment
{
"order": {
//Order Info goes here
},
"packages": [
//Collection of all packages in this order
{
"package": {
//Package info goes here
},
"items": [
//Collection of all items in this package
{
"item": {
//Item info goes here
}
}
]
}
]
}
]
}X12 Connector and DataSense
When working with the X12 connector in Anypoint Studio, the DataSense feature of the studio can greatly simplify the mapping process. A properly configured X12 connector in Anypoint Studio can show the output structure of x12:read operation using DataSense.
In our use case, as we are working on v004010/856 documents, we see the structure of the message in the Transform Message’s Input section. The 856 structure contains a “Detail → 010_HL_Loop” element that will hold an array of all HL segments parsed from the input X12 document.
The Loop is a flat array that we need to translate into the hierarchical shipments data. Each HL segment has fields representing hierarchical level information.
X12 to JSON Transformation with DataWeave
In most of the array iteration scenarios, the `map` function is sufficient for transforming data.
map(Array<T>, (item: T, index: Number) -> R): Array<R>While `map` allows us to iterate through a collection, it doesn’t let us access the collection being built.
In our case, we need a stateful iteration so that we know the position of the current record in the hierarchy when compared to previous records and to accommodate that in the final output. The other way to iterate through a collection is to use the `reduce` function.
reduce(Array<T>, (item: T, accumulator: T) -> T): T | NullUnlike the `map` function, the `reduce` function allows accessing the accumulated values during the iteration. Every iteration can prepare a new accumulator which replaces the earlier one. This makes it much more powerful than `map` as we not only get a way to access previously transformed elements but also to modify them as needed.
Here is a DataWeave script that will iterate over 856 transactions in the X12 document and prepare the required JSON structure.
%dw 2.0
output application/json
//Example shipment object structure
var shipment = {
shipment: {},
orders: [
packages:[
items: []
]
]
}
---
//Iterate over each 856.
payload.TransactionSets.v004010."856" map (st, index) -> (
//Reduce 010 HL Loop
st.Detail."010_HL_Loop" reduce ((item, accumulator = []) -> do {
//If new Shipment to new object otherwise get last shipment from accumulator
var lastShipment = if (item."010_HL".HL03 == "S") { //<1>
shipment: {
hierarchyId: item."010_HL".HL01
},
orders: [
packages: [
items: []
]
]
} else (accumulator[-1] default {}) //<2>
//Use update operator to add HL to appropriate last parent
var updatedShipment = lastShipment update {
case orders at .orders if(item."010_HL".HL03 == "S") -> [] //remove placeholder empty order
case orders at .orders if(item."010_HL".HL03 == "O") -> orders << { //<3> this is a new order, add in shipment orders.
order: {
hierarchyId: item."010_HL".HL01,
parentId: item."010_HL".HL02,
purchaseOrderNumber: item."050_PRF".PRF01
}, packages: []
}
case packages at .orders[-1].packages if(item."010_HL".HL03 == "P") -> packages << { //<4> new package in previous order, add with empty items
package: {
hierarchyId: item."010_HL".HL01,
parentId: item."010_HL".HL02
}, items: []
}
case items at .orders[-1].packages[-1].items if(item."010_HL".HL03 == "I") -> items << { //<5< new item to previous package
item: {
hierarchyId: item."010_HL".HL01,
parentId: item."010_HL".HL02,
itemNumber: item."020_LIN".LIN03
}
}
}
---
//replace last shipment object with updated one
(accumulator - lastShipment) << updatedShipment //<6>
})
)
<1>: This is where we build our Shipment Object with whatever information needed from X12 document. In the above script, we just initialized an empty skeleton object for demo.
<2>: When iterating over a non-shipment entry in the loop, we just pick the last shipment so we can add order/package/item into it.
<3>: We reached a new Order for the last Shipment. Here we map the loop item to a required Order structure and add it into the Orders collection on the last Shipment.
<4>: We reached a new Package for the last Order. We map the loop item into a Package and add it into the packages collection for last Order.
<5>: A leaf of our hierarchy, we reached a new Package Item. We map the loop item to a Package Item and add it into the last Package of the last Order.
<6>: Finally, we remove the previously built last Shipment instance from the accumulator and replace it with the Shipment modified to add orders, packages, or items. The end result of the reduce operation will be this modified accumulator.
A sample output for above script may look like this:
[
{
"shipment": {
"hierarchyId": "1"
},
"orders": [
{
"order": {
"hierarchyId": "2",
"parentId": "1",
"purchaseOrderNumber": "4NY1IIZE"
},
"packages": [
{
"package": {
"hierarchyId": "3",
"parentId": "2"
},
"items": [
{
"item": {
"hierarchyId": "4",
"parentId": "3",
"itemNumber": "9780815345558"
}
}
]
}
]
}
]
}
]
With that, we built our hierarchical data structure in JSON. It should be easy to complete the data mapping for individual components and accomplish what your business requires.
JSON to X12 Transformation with DataWeave
Consider that the same JSON file is sent by your internal APIs for delivering to your trading partners in X12 format. How can we rebuild the X12 850 transaction set from it?
The following DataWeave script should let us rebuild that X12 structure.
%dw 2.0
output application/json
fun toShipment(rec) = {
"010_HL": {
"HL01": rec.shipment.hierarchyId,
"HL03": "S"
}
// Any other Shipment related segment mapping goes here
}
fun toOrder(s, o) = {
"010_HL": {
"HL01": o.order.hierarchyId,
"HL02": o.order.parentId,
"HL03": "O"
},
"050_PRF": {
"PRF01": o.order.purchaseOrderNumber
}
// Any other Order related segment mapping goes here
}
fun toPackage(o, p) = {
"010_HL": {
"HL01": p.package.hierarchyId,
"HL02": o.order.hierarchyId,
"HL03": "P"
}
// Any other package related segment mapping goes here
}
fun toItem(p, i) = {
"010_HL": {
"HL01": i.item.hierarchyId,
"HL02": p.package.hierarchyId,
"HL03": "I"
}
// Any other Item related segment mapping goes here
}
---
{
"TransactionSets": {
"v004010": {
"856": payload map (st, index) -> {
"Heading": {
"020_BSN": {
"BSN01": "00",
"BSN02": "020201229153948475",
"BSN03": now(),
"BSN04": 56388000,
"BSN05": "0001"
}
},
"Detail": {
"010_HL_Loop": st reduce (ship, accShip=[]) -> do{
var shipUpd = accShip << toShipment(ship)
var ord = ship.orders reduce (order, accOrd = shipUpd) -> do{
var ordUpd = accOrd
<< toOrder(ship, order)
var pack = order.packages reduce (package, accPkg = ordUpd) -> do {
var packUpd = accPkg << toPackage(order, package)
var items = package.items reduce (item, accItems = packUpd) ->
accItems << toItem(package, item)
---
items
}
---
pack
}
---
ord
}
},
Summary: {
"010_CTT": {
//Sum of all shipments, orders, packages, items
CTT01: sizeOf(st.shipment) + sizeOf(flatten(st..order)) + sizeOf(flatten(st..package)) + sizeOf(flatten(st..item))
}
}
}
}
}
}Summary
X12 EDI formats may have complex data structures. Transforming data between X12 format and your internal system supported format may look challenging. DataWeave 2.0 provides many functions and modules that can help achieve the required transformation.
In the above use case, we used functions such as map, reduce, flatten, sizeOf, in addition to the update operator with pattern matching for Hierarchical transformations between X12 and JSON. Your JSON structure may not look like the one assumed in this use case, but the above DataWeave scripts should give you a good idea about how to approach building your structure.
Would you like to learn more about Hierarchical X12 EDI data mapping with DataWeave? Contact sales@avioconsulting.com to schedule a one hour consultation.