While mastering the nuances of Oracle Integration Cloud Service (ICS), I was pushing along through a transformation mapping from a fairly complex source to an equally complex target. Requirements slowly spiraled from 'simple' mappings to 'can-i-even-do-that' mappings. ICS provides a decent graphical interface for creating transformations (once you get familiar with it). It allows for an easy way to map from source to target, as long as you don't need to do anything complicated.
If you aren't familiar with ICS, check out another blog on Getting Your Feet Wet in ICS.
Although possible, it takes nearly 15 clicks (assuming you do it in the correct order) to add an if conditional similar to:
<xsl:if test="string-length($returnCode) > 0">
It's the complex scenarios and advanced functions that make transformations so powerful, has Oracle actually removed this functionality?
Of course not! The core libraries that execute a transformation are still there, and the transformations can be edited externally, e.g. in JDeveloper (or whatever xslt editing tool you wish), although it's not entirely intuitive how to make this work.
Keep in mind, that although JDeveloper makes it easy to add in complex functionality, I do suggest, to try and keep transformations simple. More about this later.
But why is this necessary?
On a recent engagement, I was tasked with merging the response data of two services, into a new enriched request. Having done this many times in SOA, I didn't think much of it.
I quickly found that the graphical interface on ICS could be a bit jumpy (especially with a large number of payload elements), and not as intuitive as I expected. The graphical interface was functional, but it only intuitively worked for my 'basic' mapping instructions, a source to target. A conditional 'if' statement requires ~15 clicks, and my requirement called for dozens of 'if' conditions, for most of my mapped attributes. There has to be a better way, I would've had to spend all day just mapping 'if's!
Doing some planning, on how to accomplish the rest of my mapping requirements, I realized I could take advantage of more advanced features in XSLT including the use of variables, selectors, and even templates, but these features don't exist in the graphical interface mapper in ICS (or maybe I couldn't find them). Although included in the graphical mapper, the if-conditions and choose-when tend to take more effort than necessary.
So why should I edit a transformation outside of ICS?
There are specific cases where you would want to edit the transformation externally.
- Some mappings can be done significantly faster in source mode. These include:
- 'If' conditionals
- 'choose-when-otherwise' conditionals
- Large sections of similar/repeating segments
- More advanced functions/transformations can also be implemented, including:
- Advanced selectors
- When a transformation has a large schema, the graphical mapper starts to require 'show more' clicks to reveal additional elements, this gets tedious rather quickly.
- JDeveloper has a decent testing harness to create sample files, and execute the transformation at design time
Tip: Editing the transformation does not enforce some rules that the graphical mapper would. These errors can be found after re-importing, or trying to activate an integration.
And why wouldn't I externally edit transformations?
While it is simple to edit a transformation outside of the ICS graphical mapper, there are reasons you shouldn't do it.
If kept simple, to update a mapping, requires an export of the entire integration, make the changes to the mapping, then a re-import back into ICS.
If you choose to use the 'import' function on the transformation (and not the entire integration), it locks that transformation. Once a mapping is locked (more on this below), ICS enforces some constraints
- The mapping is no longer able to be viewed or edited from the ICS portal. This makes it a little more difficult to debug when issues go wrong.
- It is not easy to add additional sources. It's easier to create a new mapping action that includes the required sources, then copy the transformation from the source of the original mapping document. Also, unless the sources were mapped in the same order, the namespace prefixes may have changed.
- Exporting / Importing tends to wreak havoc on the XML formatting, making it a little more difficult to locate changes.
- The built-in transformation tester in the graphical UI was convenient, although it didn't allow me to save any source files.
A Golden Rule
Ok, not golden, but the general rule, DO NOT edit mappings outside of ICS unless the mappings are complex and use features not available in the graphical mapper.
It is OK to use JDeveloper to update a mapping file with large copy/paste sections to save time, and run some external tests, however, if the mapping isn't complicated and doesn't need advanced features, keep it simple. There is extra overhead when debugging if the mapping file isn't accessible from the graphical ICS mapper.
Now, there are some scenarios that do require XSLT features not available from the ICS mapper: variables, advanced selectors, and templates; this allows us to meet those needs as well.
Just a few simple steps
Oracle has the Import/Export documented, but I'll cover what I did to make this work.
Design the integration
So now let's discuss a common flow of how, and when to update the mappings externally. Follow normal procedure for designing and implementing an integration, but leave the complicated mapping for later. Create the required connections and appropriate flow of the integration, laying out the different mapping elements. For the complex transformations, add some basic mappings from any source variables you need (we can fix this transformation later).
Tip: It's important to include at least one mapping from each different source so that they are added as a source reference in the XSL.
In some cases, the mappings start out simple and straightforward and slowly evolve into very complex mappings.
Export the integration
Now that the integration has been designed, we need to export it to get to the source files. You can manually export an integration from the ICS portal, however, check out the AVIO ICS Maven Plugin for a scripted solution (and follow best practices to get the code into source control!).
Finding / Accessing the XSL
If using the AVIO ICS Maven Plugin, the mapping files are going to be exported into <INTEGRATION>/src/main/iar/resources directory, otherwise, they will be in the exported artifacts under <INTEGRATION>.iar/icspackage/project/<INTEGRATION>/resources directory.
There isn't a great way to figure out which 'processor' folder the desired transformation is in, however, a search for *.xsl will get you pretty close. Another trick is to use the "search in files" feature on the name of the request or response object, e.g. "getObjectResponses" where GetObject is the ICS action. Keep in mind, they are numbered from first added to last, so some assumptions can be made on which folder they are in.
View in JDeveloper
Now that we know which transformations we want to edit, let's open them in our favorite IDE, JDeveloper (or other IDE of your choice).
Sure, you could open the xslt directly from JDeveloper, however, it is more convenient if we add the source files to a JDeveloper project, then we can utilize more features of the IDE.
Tip: This also allows to create sample request xml files, and save them for future debugging!
Create a new Application / Project
For my latest engagement, I created a new JDeveloper application, that contained a project for each of my integrations. Just use a plain java project when creating the integration projects.
Using a symlink or mklink (windows), create a link from within the project src directory, to the exported integrations resource directory that contains all of the xsl files. This allows us to keep JDeveloper project artifacts out of the integration source.
The below sample assumes the integration export is in the same folder as JDeveloperApplication.
Creating Symlink in Linux:
:~/exports$ cd JdeveloperApplication/AVIO_FTP_INTEGRATION/src
:~/exports/JdeveloperApplication/AVIO_FTP_INTEGRATION/src$ ln -s ../../../AVIO_FTP_INTEGRATION/src/main/iar/resources resources
Creating MKLink in Windows:
cd JdeveloperApplication\AVIO_FTP_INTEGRATION\src mklink /D win-resources ..\..\..\AVIO_FTP_INTEGRATION\src\main\iar\resources <<===>> D:\exports\AVIO_FTP_INTEGRATION\src\main\iar\resources
Tip: The symlink or mklink can be checked into source control. A mklink is visible under both Linux and Windows.
Now the resource files are visible in JDeveloper
Make the changes to the XSL
As you would a SOA project, open the xsl, and create your advanced transformation.
Tip: After exporting from ICS, the transformation is all on a single line. Use the 'reformat' feature in source mode to organize the transformation. Occasionally, after reformatting, JDeveloper will need to be restarted to view the transformation in 'design' mode.
Import the integration
Follow the instructions with the Maven plugin to reimport the integration, or create an archive file manually and import.
What to know, the Tips and Tricks
- After adding functionality that is not available from the graphical mapper, the mapping might show as a 'warning' or 'error', use the Import feature of the mapper to select the correct xsl file and 'Lock' the mapping from editing in ICS.
- If possible, try to keep the mappings simple, so that they don't have to be locked. This allows for a shorter debugging cycle if the mapping can be viewed in the ICS portal.
- If a map shows 'Warning', it could still work. Warnings are typically the result of using a variable in the transformation, and ICS isn't sure if it works. Either ignore these (not ideal), remove the variable, or use the import feature to lock the transformation.
- The first time you open a xslt in JDeveloper, it will likely be a really long single line. Use the 'Reformat' feature to make it readable. In some cases, JDeveloper still won't let you see the 'Design' view, just close and reopen JDeveloper to enable this view.