MuleSoft https://www.avioconsulting.com/blog%3Fbid%3D332 en Tips and Tricks for MuleSoft DataWeave: Delimited Files https://www.avioconsulting.com/blog/tips-and-tricks-mulesoft-dataweave-delimited-files <div class="avio-content"> <div class="page-title"> <div class="lead-image"> <div> <div>Blog Lead Image</div> <div> <img src="/sites/default/files/2019-07/Untitled%20design%20%281%29.jpg" width="1920" height="1080" alt="DataWeave" typeof="foaf:Image" /> </div> </div> <div class="dark-overlay"></div> </div> <div class="text-overlay"> <div class="container"> <div class="row"> <div class="col-sm-12"> <div><h2> <a href="/blog/tips-and-tricks-mulesoft-dataweave-delimited-files" hreflang="en">Tips and Tricks for MuleSoft DataWeave: Delimited Files</a> </h2> </div> </div> </div> </div> </div> </div> <div class="main-content"> <div class="container"> <div class="row"> <div class="col-sm-12 node-body"> <div> <div><p dir="ltr">MuleSoft’s expression language, <a href="https://www.mulesoft.com/integration-solutions/dataweave-integration">DataWeave</a>, is a powerful module a part of the MuleSoft platform. This tool can be very helpful, especially when working with delimited files. From my own experience, I have come across a few tips and tricks when using MuleSoft DataWeave for delimited files that I believe you will find helpful as well!</p> <p dir="ltr"> </p> <h1 dir="ltr">Using Invisible ASCII Characters As Delimiters In DataWeave</h1> <p dir="ltr">There will be use cases where you have to transform the incoming payload and output a delimited file. In most of the cases, it would be a CSV file where you have a comma (“,”) as the delimiter which can be achieved by defining the output format as <code>application/csv</code> in DataWeave.</p> <p dir="ltr">If you are to use a custom character as the delimiter in your output file, you can do so by using the <code>separator</code> parameter at the output. MuleSoft has provided a list of parameters you can use for CSV reader and writer properties that can be found <a href="https://docs.mulesoft.com/mule-runtime/4.2/dataweave-formats#format_csv">here</a>.</p> <p dir="ltr">For the purpose of this blog, we will only be focusing on two of the properties that are available. The first one that was mentioned above is the <code>separator</code> parameter, which separates records from another, and the <code>header</code> parameter, that indicates whether the first line of the output contains header names.</p> <p dir="ltr">As mentioned above this blog deals with how to use invisible ASCII characters like unit separator (<b>US</b>), record separator(<b>RS</b>), etc .., as your delimiter</p> <p dir="ltr">As shown in the example below, you pass the <code>separator</code> property and the values to be passed for this property should be in the Unicode Escape Sequence format. If you take a look at the snippet, the <code>separator</code> value is <code>\u001E</code> where <code>\u</code> is the Escape Sequence Representation and 001E” is the Hexadecimal Representation of the record separator (<strong>RS</strong>) ASCII character</p> <p dir="ltr"><b id="docs-internal-guid-6750ff24-7fff-a685-a72b-ad4c2f6e6e83"><img height="176" src="https://lh4.googleusercontent.com/280vaDS-doB5tugLqNjaVVKIH-XgBsPc4n0Np2x0YhEZzZXHU3JdE6AI8wttVp88PVc00pAuYFiOizhrva-9lzGTpzrzLyPkg4T3CsWsef4ShZb4q0IcTdiGeC2ERx4etDa80GFk" width="557" /></b></p> <p> </p> <p> </p> <p dir="ltr">The output of the above snippet, when opened as a text file, a single string with no delimiters will be shown as it is below:</p> <p dir="ltr"><b id="docs-internal-guid-626ec1e8-7fff-ebf1-c9ad-24d2916f95f5"><img height="45" src="https://lh5.googleusercontent.com/vPdWuZxH6u-h3GZ9cEJqJUacln1tcgBTw-0Z_wiUGsUy0KVXHxotHpcXGKS4S7qeZYTUCzp92Cev4-tpLNJvXQOgtrMORNG4R_tZe0iiDKS_MX9JN07ia7f9rRKxR0AMibp1qZsK" width="624" /></b></p> <p dir="ltr">But if you actually open this file in a text editor like Notepad++ or Sublime Text it will look like this:</p> <p dir="ltr"><b id="docs-internal-guid-60f72d19-7fff-3269-d00d-99ffbab57de6"><img height="70" src="https://lh5.googleusercontent.com/AQrTAvJlSIQAnaQ2kOlinh2T5KfecbFJfE1Wdmt4rycBJDqI49lQxacnaUdjdJs63NBT-HyrqOfNJzmjNPP9elujspt8LQi4FLzXD2MJNCkq36q1N4J4Y2JdT4nQc1DpWhRycJaZ" width="412" /></b></p> <p> </p> <p dir="ltr">Similarly, you can use any invisible ASCII character by passing the Unicode Escape Sequence as the value of separator. If you want to remove the header from your output you can always pass another property at the output that says <strong>header=false</strong> which will omit the header in your output like this:</p> <p dir="ltr"><b id="docs-internal-guid-0dbe8812-7fff-9a1c-6ae6-97ed44173220"><img height="199" src="https://lh3.googleusercontent.com/ePsERerSQn2LFCE1YK_jNdkTZT1Qg2wjOuwRWrt39O1TN9fd4rXlraQ9VTjLq6Ld_aijeOICm7dslRHJXYeyqjSpdeK5njy8cBi2TMONAg72bxvhcLsmemAVgrop122w8eJL0vGk" width="624" /></b></p> <p> </p> <p dir="ltr">This would output a delimited file with record separator as the delimiter and the header will be omitted meaning the data is populated from line 1 as shown here:</p> <p dir="ltr"><b id="docs-internal-guid-8be20522-7fff-d9c9-adea-18a80c7f3ce2"><img height="48" src="https://lh5.googleusercontent.com/sCEmcKselUka-HKK4hKR0eWumQ_Rah8hj9ERD3EaEp4aYuA2i1pvJSVhklyACkpEjGt7_gfLacBSdtpoQLUeOfh99XZTNrKiyPx29ZDMzdh5xsxLBp5x-JQ4k6VDbc_bPG09I99c" width="490" /></b></p> <p> </p> <h1 dir="ltr">Comparing Two Delimited Files Using DataWeave</h1> <p dir="ltr">This next tip will come in handy when comparing two delimited files using DataWeave. The use case is when there are two delimited files and certain matching keys in both the files, we then want to extract a certain value from the first file and map it to the second file.</p> <p dir="ltr">Here I have two pipe(“|”) delimited files, the first file has 4 fields <strong>Account</strong>, <strong>FirstName</strong>, <strong>LastName</strong> and <strong>AccountId </strong>while the second file has the fields <strong>Account</strong>, <strong>FirstName</strong>, <strong>LastName</strong>, and <strong>DOB</strong>.</p> <p dir="ltr">This use case will be generating a JSON output out of the data in these two files. If the <strong>Account</strong>, <strong>FirstName</strong>, and <strong>LastName</strong> fields in the second file match the values in the first file, then we need to extract the <strong>AccountId</strong> field from the second file.</p> <p dir="ltr">We will be creating a map with the data in the first file where the key will be <strong>Account</strong>, <strong>FirstName</strong>, and<strong> LastName</strong>, and the value will be <strong>AccountId</strong>. Then, we loop through the second file where we form a string by appending <strong>Account</strong>, <strong>FirstName</strong>, and <strong>LastName</strong> and we will lookup this key in the map we formed out of the first file.</p> <p dir="ltr"><img alt="DelimitedFile" data-entity-type="file" data-entity-uuid="d5100873-f4f7-4902-8611-141d75e289f1" src="/sites/default/files/inline-images/Code%20Snippet%201.png" /></p> <p dir="ltr"> </p> <p dir="ltr">In the snippet above, the first step is creating an empty variable named <strong>dataInFirstFile</strong> with java as the output would create an empty <strong>LinkedHashMap</strong>.</p> <p dir="ltr">The next step is reading the data from the file and saving the data in the file to the variable <strong>file1data</strong>.</p> <p dir="ltr">Once we have the file data in the file1data variable we go to DataWeave and we split the data in the file by new line character (“\n”) which will make it an array of string with each element being each line of the file. Then we split each element in the array with the delimiter and remove the first line of the file (Headers) using the <strong>splitAt</strong> function from array in DataWeave 2.2.0. We are then able to loop through the array of records and split each line by the delimiter,, in this case, it’s “|” so that would output an array of arrays which would look like this:</p> <p dir="ltr"><img alt="Code" data-entity-type="file" data-entity-uuid="7f17be06-5e92-4746-b2da-a842403ce530" src="/sites/default/files/inline-images/Code%20Snippet%202.png" /></p> <p dir="ltr"> </p> <p dir="ltr">So I am looping through this array using “map” and “$” in this case would be each array so I pick the values by their index and form a map of key-value pairs and as we loop through each iteration we add the values to the <strong>dataInFirstFile</strong> variable and the output is also stored in <strong>dataInFirstFile</strong> variable.</p> <p dir="ltr">Next step is reading the data from the second file and we will follow the same approach, rip off the header and split data into array of arrays as above and we will make a key with the <strong>Account</strong>, <strong>FirstName</strong>, <strong>LastName</strong> fields in the second file and lookup this key in the map generated out of the first file which is stored in the <strong>dataInFirstFile </strong>variable. The below snippet displays how this is done:</p> <p dir="ltr"><img alt="Code3" data-entity-type="file" data-entity-uuid="3a03e7eb-c15f-4ce9-9998-5de9a9c6120c" src="/sites/default/files/inline-images/Code%20Snippet%203.png" /></p> <p dir="ltr"> </p> <p dir="ltr">This will output a JSON with <strong>AccountId</strong> field computed from the first file as shown below:</p> <p dir="ltr"><img alt="Code4" data-entity-type="file" data-entity-uuid="2a6c0577-b6dd-49de-9ecf-49f9302dce14" src="/sites/default/files/inline-images/Code%20Snippet%204.png" /></p> <p dir="ltr"> </p> <p dir="ltr">Now you know how to use ASCII characters as delimiters and how to compare two delimited files all while using MuleSoft DataWeave. These are just a couple of tricks that I use when working with DataWeave, let me know what some of your favorites are in the comments below!</p> </div> </div> </div> </div> </div><div class="container comment-container"> <div class="text-align-center"><h2>Join the Conversation</h2> </div> <div class="row"> <div class="col-sm-12"> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=751&amp;2=comment_node_blog&amp;3=comment_node_blog" token="SpU31idvgqUNddsUOdN3KjRFoybO1ZopkGkyH0vAWaA"></drupal-render-placeholder> </div> </div> </div> <div class="container"> <div class="row"> <div class="col-sm-12 blog-tags"> <div> <div><a href="/blog%3Fbid%3D332" hreflang="en">MuleSoft</a></div> </div> </div> </div> </div> <div><drupal-render-placeholder callback="Drupal\block\BlockViewBuilder::lazyBuilder" arguments="0=views_block__blogs_block_4&amp;1=default&amp;2=en" token="jBjf0n8iki9sjPwElNYA4B92vVETYVFMv7O9pm9FdHQ"></drupal-render-placeholder></div> <div> <div>About the Author</div> <div></div> </div> </div> </div> Thu, 25 Jul 2019 15:48:47 +0000 Sai Veerepalli 751 at https://www.avioconsulting.com https://www.avioconsulting.com/blog/tips-and-tricks-mulesoft-dataweave-delimited-files#comments 3 Takeaways from MuleSoft CONNECT 2019 https://www.avioconsulting.com/blog/3-takeaways-mulesoft-connect-2019 <div class="avio-content"> <div class="page-title"> <div class="lead-image"> <div> <div>Blog Lead Image</div> <div> <img src="/sites/default/files/2019-05/IMG_6677.jpg" width="4032" height="3024" alt="MuleSoft" typeof="foaf:Image" /> </div> </div> <div class="dark-overlay"></div> </div> <div class="text-overlay"> <div class="container"> <div class="row"> <div class="col-sm-12"> <div><h2> <a href="/blog/3-takeaways-mulesoft-connect-2019" hreflang="en">3 Takeaways from MuleSoft CONNECT 2019</a> </h2> </div> </div> </div> </div> </div> </div> <div class="main-content"> <div class="container"> <div class="row"> <div class="col-sm-12 node-body"> <div> <div><p dir="ltr">This was my third year attending <a href="https://connect.mulesoft.com/events/connect/atlanta">MuleSoft CONNECT</a> since I joined AVIO, and every year it has gotten better and better. I always look forward to attending the event and learning about new things MuleSoft has planned for the future and hearing how organizations are using MuleSoft to solve their business challenges.<br />  </p> <p dir="ltr">This year was no different, so I’d like to share my 3 big takeaways from this year's conference in Atlanta.<br />  </p> <h2>1. The Keynote: tips to become a more modern organization</h2> <p dir="ltr">For this year's keynote, MuleSoft Founder, <a href="https://www.linkedin.com/in/ross-mason/">Ross Mason</a>, spoke about architecting for innovation and agility. He highlighted MuleSoft’s vision for a connected future and the importance of API’s and the application network to today's enterprises.<br />  </p> <p dir="ltr">Ross, then brought Former McDonald’s Global CIO, <a href="https://www.linkedin.com/in/frank-liberio-b9b4304/">Frank Liberio</a>, to the stage to talk about <a href="https://www.mulesoft.com/webinars/api/global-supply-chain-mcdonalds">how they used APIs and MuleSoft to overcome real problems</a>. He shared some best practices on how enterprises could accelerate digital transformation and become a more modern organization.<br />  </p> <h2>2. The vision and roadmap for the Anypoint Platform</h2> <p dir="ltr">Every year that I attend CONNECT, I look forward to hearing what new and exciting things MuleSoft has planned for the coming year. This year during the Vision and Roadmap session, <a href="https://www.linkedin.com/in/markdao/">Mark Dao</a>, Chief Product Officer, shared what we could expect to see in the IT space in the coming year, which included:<br />  </p> <ul><li>New templates to be released for Flow Designer.</li> <li>API Community Manager: an extension to Anypoint Exchange to support community members with developer forums, chat, and case management.</li> <li>The addition of Government Cloud, which is a FedRAMP compliant environment.</li> <li>Enhancements to connectors and the release of new connectors.</li> </ul><p dir="ltr"> </p> <h2>3. Technical breakout sessions</h2> <p dir="ltr">When I was looking at the agenda before the conference, I marked 3 technical breakout sessions that I knew I was going to attend.<br />  </p> <p dir="ltr">This year, I was really looking forward to the “Salesforce Integration Best Practices” breakout session because I had written a previous blog post on the same subject, <a href="https://www.avioconsulting.com/blog/how-integrate-salesforce-mulesoft">How to Integrate Salesforce with MuleSoft</a>.<br />  </p> <p dir="ltr">The other 2 breakout sessions that had caught my attention included “<em>Accelerating Digital Transformation with MuleSoft</em>” and “<em>How Leading Enterprises are Monitoring and Troubleshooting in Hybrid World</em>”. These are 2 key areas that I am always focusing on for AVIO customers.<br />  </p> <p dir="ltr">Here are some key insights I took away from each of the 3 breakout sessions:<br />  </p> <h3>Salesforce Integration Best Practices</h3> <ul><li>The MuleSoft approach integrating with Salesforce</li> <li>Demo of how to use APIs and MuleSoft can be effective in initiatives like Customer 360</li> <li>Using Heroku to deploy, run and manage applications</li> </ul><p dir="ltr"> </p> <h3>Accelerating Digital Transformation with MuleSoft</h3> <ul><li>Effective application networking in a complex ecosystem</li> <li>How different domains carry different definitions for an API layered architecture</li> <li>Integrations to legacy systems and patterns</li> </ul><p dir="ltr"> </p> <h3>How Leading Enterprises are Monitoring and Troubleshooting in Hybrid World</h3> <ul><li>How to leverage custom dashboards</li> <li>The best usage of Anypoint Monitoring</li> <li>Demo of how easy debugging is with Anypoint Monitoring</li> <li>Logging best practices</li> </ul><p dir="ltr"> </p> <p dir="ltr">MuleSoft CONNECT is something that I always look forward to every year when the dates are announced. Though the event was much smaller this year since they have broken the larger conference into multiple events, MuleSoft still delivered. From meeting all the great people at the networking events and AVIO’s booth, to holding insightful conversations about the current IT landscape, MuleSoft CONNECT 2019 was a highlight for the year and I am already looking forward to MuleSoft CONNECT 2020.<br />  </p> </div> </div> </div> </div> </div><div class="container comment-container"> <div class="text-align-center"><h2>Join the Conversation</h2> </div> <div class="row"> <div class="col-sm-12"> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=746&amp;2=comment_node_blog&amp;3=comment_node_blog" token="oShllJ2yE5as5E6a9kEGuaWtVMIZT4-h4JZPOclIY9w"></drupal-render-placeholder> </div> </div> </div> <div class="container"> <div class="row"> <div class="col-sm-12 blog-tags"> <div> <div><a href="/blog%3Fbid%3D332" hreflang="en">MuleSoft</a></div> </div> </div> </div> </div> <div><drupal-render-placeholder callback="Drupal\block\BlockViewBuilder::lazyBuilder" arguments="0=views_block__blogs_block_4&amp;1=default&amp;2=en" token="jBjf0n8iki9sjPwElNYA4B92vVETYVFMv7O9pm9FdHQ"></drupal-render-placeholder></div> <div> <div>About the Author</div> <div></div> </div> </div> </div> Thu, 23 May 2019 15:44:50 +0000 Karthik Dega 746 at https://www.avioconsulting.com https://www.avioconsulting.com/blog/3-takeaways-mulesoft-connect-2019#comments Solve Your Integration Headaches with MuleSoft https://www.avioconsulting.com/blog/solve-your-integration-headaches-mulesoft <div class="avio-content"> <div class="page-title"> <div class="lead-image"> <div> <div>Blog Lead Image</div> <div> <img src="/sites/default/files/2019-05/iStock-1081869346.jpg" width="2121" height="1414" alt="3 people looking at a computer screen " typeof="foaf:Image" /> </div> </div> <div class="dark-overlay"></div> </div> <div class="text-overlay"> <div class="container"> <div class="row"> <div class="col-sm-12"> <div><h2> <a href="/blog/solve-your-integration-headaches-mulesoft" hreflang="en">Solve Your Integration Headaches with MuleSoft </a> </h2> </div> </div> </div> </div> </div> </div> <div class="main-content"> <div class="container"> <div class="row"> <div class="col-sm-12 node-body"> <div> <div><p dir="ltr">Part of being in business today is using an array of technological tools. While tech can be used to make businesses more efficient and profitable, as more and more products are added to a company’s tech arsenal, the more you’ll face integration issues.</p> <p dir="ltr"> </p> <p dir="ltr">In a <a href="https://www.mulesoft.com/press-center/technology-trends-2019-connectivity-benchmark">2019 MuleSoft survey</a>, nearly half of companies said they use more than 1,000 different applications across their business, but only 29% of them are integrated together. That leaves important data trapped in silos, causing serious opportunity costs. And it creates an unsustainable workload for the IT department, who has a growing list of products they have to provide troubleshooting and fixes for—both on their own, and in their connections to each other.</p> <h1 dir="ltr"> </h1> <h1 dir="ltr">Why Modern Integration is Necessary</h1> <p dir="ltr">Having various products and systems disconnected is costing your business in ways you can’t necessarily see.</p> <p dir="ltr"> </p> <p dir="ltr">For example, an e-commerce business that lacks real-time access to inventory data risks disappointing customers and losing business by taking orders they can’t fill. When you can’t see the big picture with the data you have, you can’t make the smartest decisions in choosing products and services for your market. And when your IT department is bogged down with more service requests than they can handle, it slows your whole operation down and hurts the company’s overall productivity.</p> <p dir="ltr"> </p> <p dir="ltr">Your company may be hesitant to embark on an integration project because you know it can be a big undertaking, but the sooner you embrace the investment in integration, the more time, energy, and money you’ll save.</p> <h1 dir="ltr"> </h1> <h1 dir="ltr">4 Integration Challenges to Be Aware Of</h1> <p dir="ltr">To make sure your integration project goes smoothly, you should be aware of some of the common challenges that arise, so you can address them head-on.</p> <h3 dir="ltr"> </h3> <h3 dir="ltr">Internal Resistance</h3> <p dir="ltr">Any project that requires change will inevitably meet resistance. People get used to doing things a certain way and having to learn a whole new process is a hard sell. Integration technology projects often include an internal political battle to get important stakeholders on board. Convincing people that the benefits will outweigh their fears and that their concerns will be addressed is an important part of getting any digital integration project off the ground.</p> <h3 dir="ltr"> </h3> <h3 dir="ltr">Cost</h3> <p dir="ltr">Integration can help you both save and make more money, but before you can get to that point, you have to spend money. <a href="https://sdtimes.com/developers/solving-internet-things-integration-problem/">Gartner has predicted</a> that half the cost of implementing Internet of Things (IoT) technology in businesses will go towards integration efforts. And it’s not just an issue for IoT,  Gartner has also predicted that through 2020, half of the time and cost involved in building a digital platform for businesses will be for integration. Any integration efforts your company takes on will require having an adequate budget, but the cost shouldn’t stop you from taking the plunge.</p> <h3 dir="ltr"> </h3> <h3 dir="ltr">Complexity</h3> <p dir="ltr">The more disconnected apps your company depends on, the bigger a challenge it will be to get them all working together seamlessly in the future. Most businesses are moving towards having more tech products they rely on, rather than less. Without the right integration solution, compatibility issues will only get more complicated and unsustainable. But if you approach integration with an eye toward the long term, you can ensure you build a system that makes adding future apps to your company’s processes easier.</p> <h3 dir="ltr"> </h3> <h3 dir="ltr">Security</h3> <p dir="ltr">Security is too important to treat as an afterthought. Choosing the wrong integration tool can potentially increase your risk to security threats. <a href="https://www.cmswire.com/information-management/7-considerations-for-its-never-ending-integration-challenges/">70% of organizations</a> said that their integration tool increased data security and compliance risks. With the right approach, you can achieve integration without creating new security risks.</p> <h1 dir="ltr"> </h1> <h1 dir="ltr">How to Overcome Common Integration Challenges</h1> <p dir="ltr">When you know what difficulties to be aware of in advance, you can start your integration project more prepared to address them head-on. With extensive experience working on tech integration projects, AVIO has adopted some best practices for making sure integration solves a company’s primary tech problems, without adding new ones to deal with.</p> <h3 dir="ltr"> </h3> <h3 dir="ltr">Start with the business problem.</h3> <p dir="ltr">Every business is different, there’s no one-size-fits-all integration solution. Before you can develop the right integration strategy, you need to clarify the specific problems you want it to solve.</p> <h3 dir="ltr"> </h3> <h3 dir="ltr">Pick the right integration technology.</h3> <p dir="ltr">As with any problem, you need the right tool to solve it. We have experience guiding clients through MuleSoft integration projects, so you know the product offers powerful features that help solve many common integration challenges.</p> <h3 dir="ltr"> </h3> <h3 dir="ltr">Develop a long-term strategy.</h3> <p dir="ltr">You don’t just need a solution for right now, you need one that will continue to make tech compatibility issues manageable well into the future. Getting it right requires thinking long term in your plan.</p> <h3> </h3> <h3>Make a security plan part of your strategy.</h3> <p>Security needs to be a top priority in the strategy you develop. Before you start the integration process, you should establish the foundation for all integrations, which includes defining the different levels of security and access involved in each product and connection.</p> <h3 dir="ltr"> </h3> <h3 dir="ltr">Use skilled consultants to reduce your risk.</h3> <p dir="ltr">Your employees are busy and don’t want their other work disrupted by the project. By bringing in outside consultants who know how to start your MuleSoft integration off on the right foot and get the momentum going, you can reduce the impact it has on other internal projects and deadlines.</p> <h1 dir="ltr"> </h1> <h1 dir="ltr">Finding the Right Partner For Your MuleSoft Strategy</h1> <p dir="ltr">Hiring skilled consultants can help, but what’s even better is finding a partner in the process. You want a company that understands MuleSoft, is familiar with the elements of a successful integration process, and just as importantly, knows how to center your particular needs and circumstances.</p> <p> </p> <p>AVIO has proven experience with MuleSoft integration. We’ve helped clients integrate their tech and improve their processes through our unique approach. For one client, our guidance leads to <a href="https://www.mulesoft.com/sites/default/files/AVIO-Outdoor%20Cap%20Case%20Study.pdf">98% faster order processing times </a>and complete elimination of order errors. Whatever your integration problems are, we can help you solve them.</p> </div> </div> </div> </div> </div><div class="container comment-container"> <div class="text-align-center"><h2>Join the Conversation</h2> </div> <div class="row"> <div class="col-sm-12"> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=745&amp;2=comment_node_blog&amp;3=comment_node_blog" token="U66VxjKEAAIfzMXOu4KS4UoYFVQuDD6dVozyLxRyJHs"></drupal-render-placeholder> </div> </div> </div> <div class="container"> <div class="row"> <div class="col-sm-12 blog-tags"> <div> <div><a href="/blog/categories/digital-transformation" hreflang="en">Digital Transformation</a></div> <div><a href="/blog%3Fbid%3D332" hreflang="en">MuleSoft</a></div> </div> </div> </div> </div> <div><drupal-render-placeholder callback="Drupal\block\BlockViewBuilder::lazyBuilder" arguments="0=views_block__blogs_block_4&amp;1=default&amp;2=en" token="jBjf0n8iki9sjPwElNYA4B92vVETYVFMv7O9pm9FdHQ"></drupal-render-placeholder></div> <div> <div>About the Author</div> <div></div> </div> </div> </div> Wed, 22 May 2019 14:57:52 +0000 Jordan Nichols 745 at https://www.avioconsulting.com https://www.avioconsulting.com/blog/solve-your-integration-headaches-mulesoft#comments HashiCorp Vault Connector for Mule 4 https://www.avioconsulting.com/blog/hashicorp-vault-connector-mule-4 <div class="avio-content"> <div class="page-title"> <div class="lead-image"> <div> <div>Blog Lead Image</div> <div> <img src="/sites/default/files/2019-05/Mule%20%26%20Vault.png" width="1920" height="1080" alt="MuleSoft &amp; HashiCorp Vault" typeof="foaf:Image" /> </div> </div> <div class="dark-overlay"></div> </div> <div class="text-overlay"> <div class="container"> <div class="row"> <div class="col-sm-12"> <div><h2> <a href="/blog/hashicorp-vault-connector-mule-4" hreflang="en">HashiCorp Vault Connector for Mule 4</a> </h2> </div> </div> </div> </div> </div> </div> <div class="main-content"> <div class="container"> <div class="row"> <div class="col-sm-12 node-body"> <div> <div><p>HashiCorp Vault is a great place to store your secrets, but how does one quickly and easily integrate from Mule 4? Thankfully, AVIO has a Vault Connector and Vault Properties Provider. And lucky for you, both are open-source and easy to add to your project.</p> <h2>Which one should you use?</h2> <p>If you only need to retrieve values from Vault and they aren't expected to change frequently, use the properties provider. It is best to use the properties provider for database connection details as well as other connectors that require secret information at startup time.</p> <p>If you need to write secrets, handle data encryption, or retrieve secrets that change often, or are used mid-flow, use the Vault Connector. Secrets can be accessed in real-time via the connector.</p> <h2>Vault Properties Provider</h2> <p>Use the Vault Properties Provider to retrieve values from Vault. Reference values from Vault using this format:</p> <pre> ${vault::&lt;secret_engine_path&gt;/&lt;secret_path&gt;.&lt;field_name&gt;}</pre> <p>Properties are loaded from Vault when the application is started, so the application must be restarted to retrieve new values after they are updated in Vault.</p> <p>For example, one could use properties from Vault to set values in a database connection.</p> <p><img alt="Database Connection Screenshot" data-entity-type="file" data-entity-uuid="0a7dea5a-0d84-4bb1-ba4b-03f0968e8183" src="/sites/default/files/inline-images/dbConnectionScreen_0.png" /></p> <h2>Vault Connector</h2> <p>Use the Vault Connector to read or write secrets at runtime. The connector also has the ability to use the transit secrets engine to encrypt, decrypt, and re-encrypt data. </p> <p>Set up a Vault connection, then drop a Get Secret component into a flow, specify the secret path, and it's ready to go.</p> <p><img alt="Get Secret configuration" data-entity-type="file" data-entity-uuid="75eb816e-983e-4d60-883f-9c4322626ab8" src="/sites/default/files/inline-images/IMG_2345.PNG" /></p> <h2>Authenticating with Vault</h2> <p>Both of the connectors currently enable five types of authentication with Vault.</p> <ul><li>Basic Token Authentication</li> <li>TLS Authentication</li> <li>AWS IAM Authentication</li> <li>AWS EC2 Instance Metadata Authentication</li> <li>AWS EC2 Identity Document Authentication</li> </ul><h2>Where to get it</h2> <p>Find the code and installation instructions on GitHub:</p> <ul><li>Vault Connector: <a href="https://github.com/avioconsulting/mule-vault-connector">https://github.com/avioconsulting/mule-vault-connector</a></li> <li>Vault Properties Provider: <a href="https://github.com/avioconsulting/mule-vault-properties-provider">https://github.com/avioconsulting/mule-vault-properties-provider</a></li> </ul><h2>Installing the Connectors</h2> <h3>Publishing to a private exchange</h3> <p>To publish to a private exchange, some updates are necessary to the projects' <code>pom.xml</code> files and your Maven <code>settings.xml</code> file.</p> <p>In the <code>pom.xml</code> files, update the <code>groupId</code> to the Organization ID used by your organization on the Anypoint platform.</p> <p>In addition, update the <code>url</code> in the <code>distributionManagement</code> section of the pom to the following, replacing <code>${orgId}</code> with your Organization ID:</p> <pre> https://maven.anypoint.mulesoft.com/api/v1/organizations/${orgID}/maven</pre> <p>Add a <code>server</code> for the exchange repository in your Maven <code>settings.xml</code> file with <code>exchange-repository</code> as the <code>id</code> element and specify the username and password used for Anypoint Exchange.</p> <h3>Installing into local Maven repository</h3> <p>If you would like to install the connectors locally for testing before publishing to exchange, execute <code>mvn install</code>, then manually add the dependency to your project's <code>pom.xml</code>.</p> <p>For the Vault Connector, it would look like this:</p> <pre> &lt;dependency&gt; &lt;groupId&gt;${orgId}&lt;/groupId&gt; &lt;artifactId&gt;vault-connector&lt;/artifactId&gt; &lt;version&gt;0.1.1&lt;/version&gt; &lt;classifier&gt;mule-plugin&lt;/classifier&gt; &lt;/dependency&gt;</pre> <p>For the Vault Properties Provider, it would look like this:</p> <pre> &lt;dependency&gt; &lt;groupId&gt;${orgId}&lt;/groupId&gt; &lt;artifactId&gt;mule-vault-properties-providers-module&lt;/artifactId&gt; &lt;version&gt;0.1.1&lt;/version&gt; &lt;classifier&gt;mule-plugin&lt;/classifier&gt; &lt;/dependency&gt;</pre> <p> </p> <p>Now you have a great place to store your secrets and with the help of AVIO's Vault Connector and Vault Properties Provider, you are all set to quickly integrate HasiCorp Vault from Mule 4!</p> </div> </div> </div> </div> </div><div class="container comment-container"> <div class="text-align-center"><h2>Join the Conversation</h2> </div> <div class="row"> <div class="col-sm-12"> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=743&amp;2=comment_node_blog&amp;3=comment_node_blog" token="U_hRSc9DXhEsllrJMREwHo3ng_5mDOzrXvEry9V2RWY"></drupal-render-placeholder> </div> </div> </div> <div class="container"> <div class="row"> <div class="col-sm-12 blog-tags"> <div> <div><a href="/blog%3Fbid%3D332" hreflang="en">MuleSoft</a></div> <div><a href="/blog/categories/other" hreflang="en">Other</a></div> </div> </div> </div> </div> <div><drupal-render-placeholder callback="Drupal\block\BlockViewBuilder::lazyBuilder" arguments="0=views_block__blogs_block_4&amp;1=default&amp;2=en" token="jBjf0n8iki9sjPwElNYA4B92vVETYVFMv7O9pm9FdHQ"></drupal-render-placeholder></div> <div> <div>About the Author</div> <div></div> </div> </div> </div> Fri, 26 Apr 2019 15:24:39 +0000 Adam Mead 743 at https://www.avioconsulting.com https://www.avioconsulting.com/blog/hashicorp-vault-connector-mule-4#comments Documentation - You'll Thank Yourself Later https://www.avioconsulting.com/blog/documentation-youll-thank-yourself-later <div class="avio-content"> <div class="page-title"> <div class="lead-image"> <div> <div>Blog Lead Image</div> <div> <img src="/sites/default/files/default_images/network-pen-blog-3_0.jpg" width="2000" height="1333" alt="Avio Consulting" typeof="foaf:Image" /> </div> </div> <div class="dark-overlay"></div> </div> <div class="text-overlay"> <div class="container"> <div class="row"> <div class="col-sm-12"> <div><h2> <a href="/blog/documentation-youll-thank-yourself-later" hreflang="en">Documentation - You&#039;ll Thank Yourself Later</a> </h2> </div> </div> </div> </div> </div> </div> <div class="main-content"> <div class="container"> <div class="row"> <div class="col-sm-12 node-body"> <div> <div><p>Documentation. Hardly anybody wants to write it, but it is one of the most important deliverables in a software project. Without documentation, it becomes very difficult to understand why things were done the way they were and if the project is operating as expected. </p> <p>I have joined several projects where there was a lack of documentation. Sometimes you’ll be blessed with comments, but, in my experience, the only comments to be found are commented-out code. In Oracle SOA/BPM projects and Mulesoft projects, documentation is even rarer than in other projects. I’m not perfect - I’ve contributed to this problem as well. I’ve completed some code for a project and then, a month later, had to come back to the same code and figure out what the code is supposed to be doing in order to make modifications. Many developers will tell you the “code is self-documenting”, but the code doesn't show you the intent. In order to ascertain intent, one must consult documentation or requirements, and it isn’t always clear which code fulfills which requirements.</p> <p><img alt="Commented Code" data-entity-type="file" data-entity-uuid="43364b49-d141-484b-b9f0-16cc483e2808" height="131" src="/sites/default/files/inline-images/Screen%20Shot%202019-01-07%20at%201.20.40%20PM.png" width="655" class="align-left" /></p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p>We are all better than this!</p> <p> </p> <p><img alt="Mulesoft Notes Field" data-entity-type="file" data-entity-uuid="c0233439-8a75-4080-9dba-1d4fc57a2f2e" height="330" src="/sites/default/files/inline-images/Screen%20Shot%202019-01-04%20at%203.47.00%20PM.png" width="652" /><img alt="Oracle Documentation" data-entity-type="file" data-entity-uuid="99c29f98-1b67-41ac-a57a-955eb78665b5" height="256" src="/sites/default/files/inline-images/Screen%20Shot%202019-01-04%20at%203.44.24%20PM.png" width="505" /></p> <p>Save your colleagues and yourself some unnecessary time and document your code. It is possible to figure out what a chunk of code is supposed to do, but it takes much less time to read a sentence that says what the code is meant to do. The comment doesn’t have to be extremely detailed, it just needs to express the general idea. Put a comment in your Java code, fill in the Description field in your BPMN or BPEL activities, or write something in that Notes attribute on your Mulesoft message processors. Don’t leave the comments blank - you will thank yourself later!</p> </div> </div> </div> </div> </div><div class="container comment-container"> <div class="text-align-center"><h2>Join the Conversation</h2> </div> <div class="row"> <div class="col-sm-12"> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=738&amp;2=comment_node_blog&amp;3=comment_node_blog" token="Xxgpqfan-pc6CC8yR5FIbxX217LJ64swBJCwEYY6THI"></drupal-render-placeholder> </div> </div> </div> <div class="container"> <div class="row"> <div class="col-sm-12 blog-tags"> <div> <div><a href="/blog/categories/adf" hreflang="en">ADF</a></div> <div><a href="/blog%3Fbid%3D332" hreflang="en">MuleSoft</a></div> <div><a href="/taxonomy/term/365" hreflang="en">Oracle</a></div> <div><a href="/blog/categories/bpm" hreflang="en">BPM</a></div> <div><a href="/blog/categories/soa" hreflang="en">SOA</a></div> <div><a href="/blog/categories/other" hreflang="en">Other</a></div> </div> </div> </div> </div> <div><drupal-render-placeholder callback="Drupal\block\BlockViewBuilder::lazyBuilder" arguments="0=views_block__blogs_block_4&amp;1=default&amp;2=en" token="jBjf0n8iki9sjPwElNYA4B92vVETYVFMv7O9pm9FdHQ"></drupal-render-placeholder></div> <div> <div>About the Author</div> <div></div> </div> </div> </div> Mon, 07 Jan 2019 19:38:32 +0000 Adam Mead 738 at https://www.avioconsulting.com https://www.avioconsulting.com/blog/documentation-youll-thank-yourself-later#comments Mule CloudHub Logging with Custom Logger and ELK https://www.avioconsulting.com/blog/mule-cloudhub-logging-custom-logger-and-elk <div class="avio-content"> <div class="page-title"> <div class="lead-image"> <div> <div>Blog Lead Image</div> <div> <img src="/sites/default/files/2018-11/Screen%20Shot%202018-11-28%20at%2012.13.01%20AM.png" width="2542" height="1868" alt="Mule-ELK" typeof="foaf:Image" /> </div> </div> <div class="dark-overlay"></div> </div> <div class="text-overlay"> <div class="container"> <div class="row"> <div class="col-sm-12"> <div><h2> <a href="/blog/mule-cloudhub-logging-custom-logger-and-elk" hreflang="en">Mule CloudHub Logging with Custom Logger and ELK</a> </h2> </div> </div> </div> </div> </div> </div> <div class="main-content"> <div class="container"> <div class="row"> <div class="col-sm-12 node-body"> <div> <div><h2 dir="ltr"><strong>Objective:</strong></h2> <p>The objective here is to share a Robust, Consistent logging solution for Mule Cloudhub apps that are easy to maintain, visualize and track down software bugs or production issues quickly.</p> <p>Logging is a powerful aid for understanding and debugging programs run-time behavior. Logs capture and persist important data and make it available for analysis at any point in time.</p> <p>This article deals with creating a logging framework for MuleSoft apps that will suit any enterprise which is concerned about consistent and structured logging, zero log message loss and log analyzing.</p> <h2>Logging in CloudHub:</h2> <p>We don’t need to be concerned with the logging configuration inside Mule apps when we choose to deploy them to CloudHub. CloudHub writes all the logs to the console, provides a way to search by keyword and by the level and gives us an ability to change applications root log level on the fly, and to switch to debug mode and such.</p> <h3>So, why not use CloudHub’s default logging then?</h3> <p dir="ltr">Whenever Mule apps are deployed to CloudHub, logging capabilities can quickly become very limited in terms of visibility, persistence etc. especially when you have large amounts of logs that are generated by applications. Here are some things that you have to keep in mind before considering default CloudHub logging as your long-term thing.</p> <ul dir="ltr"><li>Persists logs only up to 100 MB or 30 days whichever comes first.</li> <li>After you delete applications, you lose all logs of that application.</li> <li>The only way to see logs is by logging into Anypoint platform and see CloudHub console per application.</li> <li>Cannot ship logs to your desired system easily, they can only be accessed via the REST API.</li> <li>No single place to analyze all your applications logs especially if you have API-led architecture. This can help to take some preventive measures.</li> </ul><h3><strong>So...</strong></h3> <ul dir="ltr"><li>We need a logging framework that can generate logs in a consistent and structured manner.</li> <li>Use any external log analyzer like ELK and feed all the logs to it to analyze your data in the most efficient way.</li> </ul><h3>First things first.</h3> <ul><li>Raise a support ticket with MuleSoft to disable CloudHub logging.  This is the setting that MuleSoft team has to do to in order for you to override the default CloudHub log4j2 configuration with your own custom application configuration.</li> <li>Setup Elastic stack(E.L.K) to analyze log data. Two things to consider here. <ul><li>As those three products are open source, you can use this <a href="https://elk-docker.readthedocs.io/">docker image</a> to set up your own environment and maintain all the configurations, deployments, disaster recovery, security, backup, etc.</li> <li>There is a company called <a href="https://www.elastic.co/">elastic.co</a> which takes care of all the pain in managing instances, disaster recovery, backup, etc. and they provide a way to set up stack on-premise or on their managed cloud.</li> </ul></li> </ul><pre dir="ltr"> <strong>Note: </strong>When you enable custom logging in CloudHub, You can change the application's loggers log level on the fly via CloudHub API. But you cannot see a place in UI to do so. Hope it get fixed soon.</pre> <h2>Design a Custom logger for your Mule applications:</h2> <p>The default Mule logger is simple to use for logging but can be too simple because it doesn’t enforce any standards or best practices.  It also doesn’t provide a way to generate structured or consistent log messages. Designing a custom logger is the best solution here to generate structured logs and to enforce standards on the log data to be generated and of course, for seamless developer usage. The idea of this module should be to generate consistently structured logs in JSON format.</p> <p>It would be very good to provide a mule custom module for a developer to use for logging, requires specific information, and also be extended with additional information as key-value pairs, to log at a certain point, in a neat GUI. Then it’s the custom module’s job to take all the information, convert that data to JSON and log.</p> <p dir="ltr">Here is the look of Mule Custom Logger developed by AVIO to use for any log purposes.</p> <p dir="ltr"><img alt="custom-logger" data-entity-type="file" data-entity-uuid="7f901a09-e7fb-4cc8-a95b-3f62d76ad899" src="/sites/default/files/inline-images/custom-logger.png" /></p> <p dir="ltr"> </p> <p dir="ltr"> </p> <p dir="ltr">And this module will produce a JSON like this</p> <table><colgroup></colgroup><tbody><tr><td> <pre> { "app_name": "sample", "app_version": "1.0.0", "env": "dev", "ext": { "additionalKey1": "additionalValue1", "additionalKey2": "additionalValue2" }, "location": { "component": "avio-core:custom-logger", "file_name": "sample.xml", "line_in_file": "16", "location": "sampleFlow/processors/1", "root_container": "sampleFlow" }, "log": { "category": "com.avio", "correlation_id": "33af6200-018b-42af-8ec1-ebdbf29c0505", "level": "INFO", "message": "This is nothing but a sample log message", "tracePoint": "START" }, "thread": "[MuleRuntime].cpuLight.23: [sample].sampleFlow.CPU_LITE @27346628", "timestamp": "2019-05-17T15:06:34.592Z" }</pre> </td> </tr></tbody></table><p>This logger has been open-sourced under BSD 2-Clause "Simplified" License in GitHub. You can find it <a href="https://github.com/avioconsulting/mule-custom-logger">here</a>.</p> <p>We now have to feed this JSON to the ELK stack where Logstash will read this JSON message, parse it and ships to elasticsearch. Here is the high-level overview of ELK Stack. Now we have a great UI and a mechanism to generate logs in a consistent structured manner.  Also if we ever need to change the implementation of how and where we are logging to, the custom module can be updated and each project simply needs to update to the latest version of the module.</p> <p dir="ltr"><img src="https://lh6.googleusercontent.com/PhuGjzxicTZkcrrmYDvoJLIBlIEsjvdKaBoUeec1gYiQU6Xg7pimtNMFR_RSzua9WvxS6ExuMVRZh181MxEvbns1XuyxHLPBCxh57vrohQRteKDEN-Ee6d67lXGNusPhZPHsGEPn" /></p> <p>When you disabled CloudHub logging for an application, There are multiple ways to ship logs from CloudHub to any external system(Logstash in our case). Here are some.</p> <ul dir="ltr"><li>Use Log4j2 appenders like socket appender to post all logs to a specified destination.</li> <li>Use MuleSoft provided <a href="https://anypoint.mulesoft.com/exchange/portals/anypoint-platform/f1e97bc6-315a-4490-82a7-23abe036327a.anypoint-platform/cloudhub-api/1.0.5/pages/home/">CloudHub API</a> to retrieve logs periodically and send to your destination.</li> <li> <div>[Recommended]: Use a log4j2 appender to push all your CloudHub logs to Amazon SQS queue and have Logstash read messages from it. This way, we can scale SQS queue for high availability when there is huge log flow and we can ensure zero log message loss from CloudHub to Logstash as there is a highly scalable queue in between. <a href="https://github.com/avioconsulting/log4j2-sqs-appender">Here is the code</a> for the custom log4j2 appender to push all application logs to AWS SQS queue.</div> </li> </ul><h3>Logstash Pipeline:</h3> <p>Considering that you have <a href="https://www.elastic.co/guide/en/logstash/current/setup-logstash.html">Logstash setup</a>, you can write pipelines in Logstash to read messages from the queue, parse them and send to elasticsearch.</p> <p dir="ltr">Here is an example of Logstash pipeline that reads a message from AWS SQS queue, parses each message and send to elastic to store.</p> <table><colgroup></colgroup><tbody><tr><td> <p dir="ltr">input {<br />  sqs {<br />    id =&gt; "mule-sqs"<br />    access_key_id =&gt; "&lt;&lt;aws-access-key&gt;&gt;"<br />    secret_access_key =&gt; "&lt;&lt;aws-secret&gt;&gt;"<br />    id_field =&gt; "sqs.id"<br />    md5_field =&gt; "sqs.md5"<br />    sent_timestamp_field =&gt; "sqs.sent_time"<br />    polling_frequency =&gt; 5<br />    queue =&gt; "&lt;&lt;sqs-queue-name&gt;&gt;"<br />    region =&gt; "&lt;&lt;aws-region&gt;&gt;"<br />    codec =&gt; "plain"<br />  }<br /> }<br /><br /> filter {<br />  grok {<br />    match =&gt; {<br />      message =&gt; "%{WORD}%{SPACE}%{TIMESTAMP_ISO8601:logdate}\s\[%{GREEDYDATA:thread}]%{SPACE}%{GREEDYDATA}: %{GREEDYDATA:message}"<br />    }<br />    overwrite =&gt; ["message"]<br />  }<br />  date {<br />    match =&gt; ["logdate", "yyyy-MM-dd HH:mm:ss,SSS", "ISO8601"]<br />  }<br />  json {<br />    source =&gt; "message"<br />  }<br /> }</p> <p dir="ltr"><br /> output {<br />  elasticsearch {<br />    hosts =&gt; "&lt;&lt;elastic-host&gt;&gt;"<br />    ssl =&gt; true<br />    manage_template =&gt; false<br />    index =&gt; "mulesoft-%{+YYYY.MM.dd}"<br />    user =&gt; "&lt;&lt;elastic-username&gt;&gt;"<br />    password =&gt; "&lt;&lt;elastic-password&gt;&gt;"<br />    codec =&gt; "plain"<br />  }<br /> }</p> </td> </tr></tbody></table><p>Here is AVIO Consulting's log4j2 custom log appender to ship all the application logs to a specified Amazon SQS queue. You can find this custom appender <a href="https://github.com/avioconsulting/log4j2-sqs-appender">here</a>. It is open-sourced as well under BSD 2-Clause "Simplified" License.</p> <p>This Logstash pipeline has 3 parts.</p> <ul dir="ltr"><li><strong>Input{}:</strong> This has some configuration to read a message from the specified AWS SQS queue.</li> <li><strong>Filter{}:</strong> This takes each log message from SQS queue, reads and parses it as JSON, creates appropriate fields based on the incoming JSON.</li> <li><strong>Output{}:</strong> This has the configuration to send all parsed messages to a specified destination. It should be the elastic endpoint in this</li> </ul><p>case.</p> <ul dir="ltr"></ul><p>As Logstash reads and parses the log data, elastic on the other hand stores all the data with the appropriate fields as we see below in Kibana.</p> <p dir="ltr"><img src="https://lh6.googleusercontent.com/K3H6Y1x53rENtsbo78zQOq8h7r7FMyjYH1PYkEeBs2ohY0TzJQDZVBYga3FM7uW0T8IuxO1p9uODjwgQXYA1cVcow9mCJFr47j-tiu0op8c4ktYxS8ZBAGmO-7-Em9uNefexnEMt" /></p> <p>In the above picture, you are seeing details about one log message in Kibana dashboard that is generated by our custom logger. You can also see all the JSON fields from the log message on the left pane.</p> <p>Note that the Logstash pipeline is reading the entire JSON log message from SQS queue and creates corresponding fields to each of the key-value pairs out of that incoming JSON.  When you have a dedicated field in elastic for each of the data points you send from your mule app, it is incredibly easy to create stunning dashboards and charts in Kibana.</p> <p>Now you can use Kibana to create charts and dashboards out of the data point fields in elastic for much better visualization of log messages. You can also configure your elastic instance to persist logs for an extended period of time as well as provide rollups and archiving options, things that aren’t possible in standard Cloudhub logging.</p> <p>We have now streamlined the generation of consistently structured log data from the Mule application and transported it to a highly available AWS SQS queue, where it was read and parsed by Logstash, and then sent to elasticsearch with all the fields for you to play with inside Kibana.</p> </div> </div> </div> </div> </div><div class="container comment-container"> <div class="text-align-center"><h2>Join the Conversation</h2> </div> <div class="row"> <div class="col-sm-12"> <div class="container blog-comments"> <div class="row"> <div class="col-sm-12"> <div class="pull-left author"> <div>Kalyan Kankanala</div> </div> <div class="comment-date"> <div>December 12, 2018</div> </div> </div> </div> <div class="row"> <div class="col-sm-12 comment-body"> <div><p>Hi Team,</p> <p>Could you please let me know what is the configuration used in log4j2.xml file . I have all the things needed from logzio and Mulesoft , however Mulesoft is throwing error LogzioAppender is not valid. Can you throw some sample .xml and pom file if any.</p> <p>Appreciate your help.</p> <p>Regards,<br /> KK</p></div> </div> </div> </div> <div class="container blog-comments"> <div class="row"> <div class="col-sm-12"> <div class="pull-left author"> <div>Chakri Bonthala</div> </div> <div class="comment-date"> <div>December 12, 2018</div> </div> </div> </div> <div class="row"> <div class="col-sm-12 comment-body"> <div><p>Hello Kalyan,</p> <p>We have a custom appender developed by AVIO to push all the mule application logs to a highly available Amazon SQS queue and we have Logstash pipelines read messages from SQS queue and send to Kibana. So, In our log4j2.xml, we have a custom appender config for AWS SQS queue.</p></div> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=737&amp;2=comment_node_blog&amp;3=comment_node_blog" token="2giSD4QVq97-bVfXya9CeHDIFdH1NitylDlDDM8jgXY"></drupal-render-placeholder> </div> </div> </div> <div class="container"> <div class="row"> <div class="col-sm-12 blog-tags"> <div> <div><a href="/blog%3Fbid%3D332" hreflang="en">MuleSoft</a></div> </div> </div> </div> </div> <div><drupal-render-placeholder callback="Drupal\block\BlockViewBuilder::lazyBuilder" arguments="0=views_block__blogs_block_4&amp;1=default&amp;2=en" token="jBjf0n8iki9sjPwElNYA4B92vVETYVFMv7O9pm9FdHQ"></drupal-render-placeholder></div> <div> <div>About the Author</div> <div></div> </div> </div> </div> Tue, 04 Dec 2018 17:16:18 +0000 Chakri Bonthala 737 at https://www.avioconsulting.com https://www.avioconsulting.com/blog/mule-cloudhub-logging-custom-logger-and-elk#comments JavaScripting in Mule 3 / 4 https://www.avioconsulting.com/blog/javascripting-mule-3-4 <div class="avio-content"> <div class="page-title"> <div class="lead-image"> <div> <div>Blog Lead Image</div> <div> <img src="/sites/default/files/default_images/network-pen-blog-3_0.jpg" width="2000" height="1333" alt="Avio Consulting" typeof="foaf:Image" /> </div> </div> <div class="dark-overlay"></div> </div> <div class="text-overlay"> <div class="container"> <div class="row"> <div class="col-sm-12"> <div><h2> <a href="/blog/javascripting-mule-3-4" hreflang="en">JavaScripting in Mule 3 / 4</a> </h2> </div> </div> </div> </div> </div> </div> <div class="main-content"> <div class="container"> <div class="row"> <div class="col-sm-12 node-body"> <div> <div><h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Overview</span></h2> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Often when learning a new tool, IDE or language depending on how curious you are or how much time you have to get to play with it you get to discover many different things about it.</span></p> <p dir="ltr">This time around when learning Mulesoft a few months ago using both 3/4 Anypoint Studios I had to learn of course the basic components and once I felt comfortable with them I started using the not so common components (based on the tutorials I did).</p> <p dir="ltr">I'll talk about the experience that I had using the scripting component using the JavaScript engine and since there's no much out there (Most examples in the documentation are about Python and Groovy) I'll share with you the few key things I found useful.</p> <p dir="ltr">The following techniques apply to both Mule 3 or 4 unless I say otherwise.</p> <h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Read the payload using the eval() JS function</span></h2> <pre> <code class="language-javascript">var payloadVar = eval('(' + payload + ')');</code></pre> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Reading the payload like this will allow you to treat it as a JavaScript object therefore use native JavaScript functions to work with objects, iterate arrays or filter them.</span></p> <p dir="ltr">For instance -&gt;</p> <pre> <code class="language-javascript">var payloadVar = eval('(' + payload + ')'); var items = payloadVar.listOfItems; var serviceItems = items.filter(function(i) {return i.itemType == 'Service'});</code></pre> <h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Log information through the script in the Mule console</span></h2> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Pretty simple, yet not documented. You can use the log.info() function to give detailed information in the console when the script is being executed.</span></p> <p dir="ltr">Like this -&gt;</p> <pre> <code class="language-javascript">var simpleString = "simple"; log.info("Hello World"); log.info(simpleString); //"simple"</code></pre> <h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Returning the payload as JSON</span></h2> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">It's a bit more critical in Mule 3 because if you don't return it as JSON and then you pass the payload for instance to the response of a call. It will be read as 'unexpected c'.</span></p> <p dir="ltr">In Mule 4 it's pretty much the same. The difference is that if you return the payload as it is it will be treated in the flow as a HashMap generated by the Nashorn JavaScript engine.</p> <pre> <code class="language-javascript">var payloadToReturn = { customerName: payloadVar.customerName, customerId: payloadVar.customerId, orderNumber: payloadVar.orderNumber, totalServicesCost: 23453, totalItemCost: 345, totalCost: 2798 } //mule 3 payload = JSON.stringify(payloadToReturn); //mule 4 payload = payloadToReturn; </code></pre> <p dir="ltr"> </p> <h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">How to deal with Flow Variables</span></h2> <h3 dir="ltr">Mule 3</h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Let's say there's a set variable outside the script that sets variable1 = "I am variable 1"</span></p> <pre> <code class="language-javascript">var simpleString = flowVars.get('variable1'); log.info(simpleString); //output: "I am variable 1" flowVars.put('variable1', "I'm variable 1 changed"); log.info(flowVars.get('variable1')); //output: "I'm variable 1 changed"</code></pre> <h3 dir="ltr">Mule 4</h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Same scenario with the set variable component</span></p> <pre> <code class="language-javascript">var simpleString = vars.variable1; log.info(simpleString); //output: "I am variable 1" vars.variable1 = "changed"; //this doesn't work log.info(flowVars.get('variable1')); //output: "I'm variable 1 changed"</code></pre> </div> </div> </div> </div> </div><div class="container comment-container"> <div class="text-align-center"><h2>Join the Conversation</h2> </div> <div class="row"> <div class="col-sm-12"> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=736&amp;2=comment_node_blog&amp;3=comment_node_blog" token="7C6oGw5WX31M_hCrsGGef3MR1nFj2wyEhWfKuOvJgxk"></drupal-render-placeholder> </div> </div> </div> <div class="container"> <div class="row"> <div class="col-sm-12 blog-tags"> <div> <div><a href="/blog%3Fbid%3D332" hreflang="en">MuleSoft</a></div> </div> </div> </div> </div> <div><drupal-render-placeholder callback="Drupal\block\BlockViewBuilder::lazyBuilder" arguments="0=views_block__blogs_block_4&amp;1=default&amp;2=en" token="jBjf0n8iki9sjPwElNYA4B92vVETYVFMv7O9pm9FdHQ"></drupal-render-placeholder></div> <div> <div>About the Author</div> <div></div> </div> </div> </div> Wed, 21 Nov 2018 22:08:26 +0000 Alvaro Zorzini 736 at https://www.avioconsulting.com https://www.avioconsulting.com/blog/javascripting-mule-3-4#comments Guide to Securing Mule 4 APIs with OAuth 2.0 and Okta https://www.avioconsulting.com/blog/secure-mule-4-apis-oauth-20-and-okta <div class="avio-content"> <div class="page-title"> <div class="lead-image"> <div> <div>Blog Lead Image</div> <div> <img src="/sites/default/files/2018-11/Okta%20MuleSoft_0.png" width="1919" height="1003" alt="Okta MuleSoft" typeof="foaf:Image" /> </div> </div> <div class="dark-overlay"></div> </div> <div class="text-overlay"> <div class="container"> <div class="row"> <div class="col-sm-12"> <div><h2> <a href="/blog/secure-mule-4-apis-oauth-20-and-okta" hreflang="en">Guide to Securing Mule 4 APIs with OAuth 2.0 and Okta</a> </h2> </div> </div> </div> </div> </div> </div> <div class="main-content"> <div class="container"> <div class="row"> <div class="col-sm-12 node-body"> <div> <div><h2><meta charset="utf-8" /><b id="docs-internal-guid-5b7eef97-7fff-b5c0-e579-176b929401e7">Introduction </b></h2> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Okta is a popular enterprise identity and access management provider, and the Anypoint API Manager enables you to govern and secure Mule APIs with a comprehensive policy library and integrated access management.  </span></p> <p style="margin:10pt 0pt 0pt"><span style="line-height:200%">I am going to guide you through the set up of Okta as the OpenID Connect (OIDC) Identity provider for your Anypoint organization so you can use its OAuth 2.0 authorization server and access management capabilities to secure your APIs.</span></p> <p style="margin:10pt 0pt 0pt"><span style="line-height:200%">OIDC is an identity layer on top of the <a href="https://oauth.net/2/">OAuth 2.0</a> protocol and I am going to cover OAuth 2.0 using the <a href="https://www.oauth.com/oauth2-servers/access-tokens/client-credentials/">client credentials grant</a>. However, this setup will let you implement other grants and add the OIDC layer.</span></p> <p style="margin:10pt 0pt 0pt"><span style="line-height:200%">According to the MuleSoft documentation, this process should work for any standard OpenID Connect Identity Provider that supports Dynamic Client Registration.</span></p> <p style="margin:10pt 0pt 0pt"><span style="line-height:200%">The diagram below gives a quick overview of the OAuth dance (OAuth authorization process) used for the client credentials grant type, which is used for machine to machine applications. With this grant type, there is no user involved in the OAuth dance. In Okta, the token URL (/token) of your authorization server is used by the client application to request the access token, and the token introspection URL (/introspect) is used internally by the API Manager policy to validate the access token. We are going to review every step in detail in the following sections.</span></p> <p dir="ltr"><img alt="oauth" data-entity-type="file" data-entity-uuid="b49c96b3-8600-41a0-9327-e715a3f393ea" src="/sites/default/files/inline-images/1_0.png" /></p> <h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Prerequisites</span></h2> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">There are few things that you need to make sure you have set up and signed up for before you get started:</span></p> <ul><li style="margin:5pt 0pt 0pt"><span style="line-height:100%">A <a href="https://www.mulesoft.com/anypoint-pricing">Platinum or Titanium subscription</a>. The feature that allows integration with an OpenID Connect Identity provider is called External Identity Management</span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">You need administrator privileges to the Master Organization in Anypoint Platform. If you don’t have Admin privileges, you can sign up for a trial account here: <a href="https://anypoint.mulesoft.com/login/#/signup">https://anypoint.mulesoft.com/login/#/signup</a></span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">An Okta Account. You can get an Okta Developer Account here: <a href="https://developer.okta.com/signup/">https://developer.okta.com/signup</a></span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">An API deployed, active in the API Manager, and published in Exchange</span></li> </ul><p dir="ltr"><img alt="okta" data-entity-type="file" data-entity-uuid="eec897fb-6b57-4fa0-bb1a-069e7b39167c" src="/sites/default/files/inline-images/2_0.png" /></p> <h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">What We Are Covering</span></h2> <ul dir="ltr"><li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Set up Token Introspection Client app in Okta </span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Generate SSWS token in Okta.</span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Register Okta as an OpenID Connect Identity Provider / OAuth 2.0 Authorization Server in Anypoint Platform.</span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Register your API in Okta and add the client credentials grant.</span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Add a custom scope in Okta and assign it to your application. </span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Apply the OAuth 2.0 security policy to the Mule API.</span></li> </ul><h3 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Set up Token Introspection Client app n Okta</span></h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">In Okta, add a new application by going to the Applications menu on the top of the screen, click on Add Application, select Web, and click Next. In the Application Settings form, enter the application name, check the Client Credentials Grant type, and click Done. </span></p> <p dir="ltr"><img alt="image 3" data-entity-type="file" data-entity-uuid="d5ba7ec2-623c-4bf4-81e2-ee8bc3894483" src="/sites/default/files/inline-images/3.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">From the next screen, copy the Client ID and Client Secret in a notepad for later use.</span></p> <p style="margin:10pt 0pt 0pt"><span style="line-height:200%">I called the application “Token Introspection Client” to match the name in the Anypoint’s Dynamic Client Registration form. This will allow Anypoint’s API Manager to validate the OAuth 2.0 access token using the information provided by Okta’s <a href="https://developer.okta.com/docs/api/resources/oidc#introspect">introspection endpoint</a>.</span></p> <h3 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Generate SSWS token in Okta</span></h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">In Okta, go to the API menu, select Authorization Servers, go to the Tokens tab, click Create Token, and copy the token in a notepad for later use. This is the Single Sign-On Web System (SSWS) token.    </span></p> <p dir="ltr"><img alt="image 4" data-entity-type="file" data-entity-uuid="0247e231-1b4c-42b6-9e92-15ab476eb0ac" src="/sites/default/files/inline-images/4.png" /></p> <h3 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Register Okta as an OpenID Connect Identity Provider / OAuth 2.0 Authorization server in Anypoint Platform</span></h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Go to the Anypoint Platform’s menu on the left, select Access Management under Management Center, then click on External Identity.  Expand Client Management and select OpenID Connect Dynamic Client Registration.</span></p> <p dir="ltr"><img alt="image 5" data-entity-type="file" data-entity-uuid="0e35e200-a49b-4097-a070-58d7e52ffc4c" src="/sites/default/files/inline-images/5.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Fill out the Dynamic Client Registration form: </span></p> <p dir="ltr"><img alt="image 6" data-entity-type="file" data-entity-uuid="cd46485f-7647-4c58-b460-c285f1ebb1a4" src="/sites/default/files/inline-images/6.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">The Client ID and Client Secret come from the Token Introspection Client app we registered in Okta in the previous section. The Authorization Token is the SSWS we also created in the previous section, and the rest of the information comes from the authorization server metadata in Okta. </span></p> <p style="margin:10pt 0pt 0pt"><span style="line-height:200%">To get the server’s metadata, go to Okta’s API menu and select Authorization Servers. Click on the default server (or the server you are using) and follow the metadata URI link to get the URLs needed and save the form.</span></p> <p dir="ltr"><img alt="image 7" data-entity-type="file" data-entity-uuid="cb9d0046-e8aa-4cc3-9a69-d71d473e704b" src="/sites/default/files/inline-images/7_0.png" /></p> <h3 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Register your API in Okta and add the client credentials grant</span></h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Go to your API in Anypoint Exchange and from the vertical ellipses select Request Access.</span></p> <p dir="ltr"><img alt="image 8" data-entity-type="file" data-entity-uuid="2d0c2b50-7a3c-4f72-9913-96c4226afa07" src="/sites/default/files/inline-images/8_0.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">In the Request API access window, click on Create a new application. In the Create new application form, enter your application’s name, select Authorization Code Grant because you have to select a grant (later we’ll add the Client Credentials Grant in Okta). The OAuth 2.0 redirect URI is not needed for the Client Credentials grant flow, but I added it to try the Authorization Code grant flow later. </span></p> <p dir="ltr"><img alt="image 9" data-entity-type="file" data-entity-uuid="2ff7b0a7-d7cd-42b7-b9c3-ab2ed8e856a4" src="/sites/default/files/inline-images/9_0.png" /></p> <p style="margin:11pt 0pt 0pt"><span style="line-height:200%">In the following Request API access window select the API Instance and click Request API Access. This step will create your API in Okta and will return its client id and Client secret. Copy the Client id and Client secret for later use.</span></p> <p dir="ltr"><img alt="image 10" data-entity-type="file" data-entity-uuid="e8b16c0d-6bf7-44b6-bd8c-c9af195cd574" src="/sites/default/files/inline-images/10_0.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">In Okta, go to Applications and you will see your application listed there. Click on the application’s settings gear icon, select Assign to Groups, assign to Everyone and click Done.</span></p> <p dir="ltr"><img alt="image 11" data-entity-type="file" data-entity-uuid="e48be003-d364-4347-b577-de76782fe110" src="/sites/default/files/inline-images/11_0.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Then click on your application (Products API in my case), go to the General tab to display the General Settings, click on Edit, check the Client Credentials box and save it. </span></p> <h3 style="margin: 20pt 0pt 0pt;"><span style="line-height:107%">Add a custom scope in Okta and assign it to your application</span></h3> <p style="margin:1pt 0pt 0pt">In Okta, go to the API menu and select Authorization Servers, click on the link to your server, and go to the Scopes tab. Click Add Scope and fill out the form to create “myscope.” We are going to authorize this scope in the API Manager when we apply the security policy.</p> <p style="margin:10pt 0pt 0pt">Let’s note that the default access policies in Okta allow any registered app to use any scope and any grant assigned to the app to request tokens. You can add and customize access policies, but the administration of Okta is out of the scope of this blog.</p> <p dir="ltr"><img alt="image 12" data-entity-type="file" data-entity-uuid="cc52730d-1167-417a-83d0-fd02bbe2ee6e" src="/sites/default/files/inline-images/12_0.png" /></p> <h3 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Apply the OAuth 2.0 security policy to the Mule API</span></h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">In Anypoint, go to your API settings in the API Manager and click on Policies.</span></p> <p dir="ltr"><img alt="image 13" data-entity-type="file" data-entity-uuid="7e659f5c-feb8-48b5-8725-f1829451ada5" src="/sites/default/files/inline-images/13_0.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Click on Apply New Policy, in the Categories drop-down menu select Security, expand OpenId Connect access token enforcement, and select the version for your application. Click on Configure Policy.</span></p> <p dir="ltr"><img alt="image 14" data-entity-type="file" data-entity-uuid="f77bc4d6-94a1-4b10-8810-351e16b389cd" src="/sites/default/files/inline-images/14_0.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Add “myscope” and click Apply. The policy takes about a minute to become in effect. I am applying the policy to all the API methods and resources, but you can also apply more granular policies and even create a <a href="https://docs.mulesoft.com/api-manager/2.x/custom-policy-4-reference">custom policy</a> according to your project needs.</span></p> <p dir="ltr"><img alt="image 15" data-entity-type="file" data-entity-uuid="f7b2d5df-634f-4fc7-a60f-d9ef3d44e123" src="/sites/default/files/inline-images/15_0.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">The last step is to update your application’s RAML with the RAML snippet provided. </span></p> <p dir="ltr"><img alt="image 16" data-entity-type="file" data-entity-uuid="54f2b93b-bdf4-4466-aa96-ffef230c020c" src="/sites/default/files/inline-images/16_0.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">At this point, your API is protected. If you try to access it without a token you will get a 400 error "error": "Access token was not provided."</span></p> <p style="margin:10pt 0pt 0pt"><span style="line-height:200%">To get an access token you need to send a POST Request to Okta’s token URL (/token) provided by your server’s metadata when you filled out the dynamic registration form above. The POST request should be of content-type application/x-www-form-urlencoded, with the parameters grant_type=client_credentials and scope=myscope. Use basic authentication with your application’s client id as the user and client secret as the password.</span></p> <p style="margin:10pt 0pt 0pt"><span style="line-height:200%">Now you can make a call to your API by either adding the query param access_token with the access token or sending the token as a Bearer token in the Authorization header.</span></p> </div> </div> </div> </div> </div><div class="container comment-container"> <div class="text-align-center"><h2>Join the Conversation</h2> </div> <div class="row"> <div class="col-sm-12"> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=733&amp;2=comment_node_blog&amp;3=comment_node_blog" token="jlRO_cTlRnCUwaG99GecbwJ1mIgPTr2SZJQgyBQ3dTA"></drupal-render-placeholder> </div> </div> </div> <div class="container"> <div class="row"> <div class="col-sm-12 blog-tags"> <div> <div><a href="/blog%3Fbid%3D332" hreflang="en">MuleSoft</a></div> </div> </div> </div> </div> <div><drupal-render-placeholder callback="Drupal\block\BlockViewBuilder::lazyBuilder" arguments="0=views_block__blogs_block_4&amp;1=default&amp;2=en" token="jBjf0n8iki9sjPwElNYA4B92vVETYVFMv7O9pm9FdHQ"></drupal-render-placeholder></div> <div> <div>About the Author</div> <div></div> </div> </div> </div> Thu, 15 Nov 2018 14:49:42 +0000 Miguel Martinez 733 at https://www.avioconsulting.com https://www.avioconsulting.com/blog/secure-mule-4-apis-oauth-20-and-okta#comments How to Integrate Salesforce with MuleSoft https://www.avioconsulting.com/blog/how-integrate-salesforce-mulesoft <div class="avio-content"> <div class="page-title"> <div class="lead-image"> <div> <div>Blog Lead Image</div> <div> <img src="/sites/default/files/2018-09/MuleSoft%20plus%20SalesForce.png" width="1920" height="1080" alt="MuleSoft plus SalesForce" typeof="foaf:Image" /> </div> </div> <div class="dark-overlay"></div> </div> <div class="text-overlay"> <div class="container"> <div class="row"> <div class="col-sm-12"> <div><h2> <a href="/blog/how-integrate-salesforce-mulesoft" hreflang="en">How to Integrate Salesforce with MuleSoft</a> </h2> </div> </div> </div> </div> </div> </div> <div class="main-content"> <div class="container"> <div class="row"> <div class="col-sm-12 node-body"> <div> <div><h1 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Integration Patterns:</span></h1> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Salesforce is one of the most integration-friendly products available that supports real-time, asynchronous or even batch processing. This blog discusses the various options we have for integrate with salesforce rather than how each of these to be implemented.</span></p> <h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Outbound (Integrate Salesforce to Remote Systems):</span></h2> <h3 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Real Time:</span></h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Integration platforms can expose a web service using the WSDL generated by Salesforce which means integration should implement salesforce interfaces. MuleSoft does provide all the tools needed for the integration.</span></p> <p><img alt="apiFlow" data-entity-type="file" data-entity-uuid="3728e00b-0fc9-4579-aaac-edccd0d530c1" src="/sites/default/files/inline-images/image%201_0.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">In the test use case I choose, the implementation looks like the following</span></p> <p><img alt="Implementation" data-entity-type="file" data-entity-uuid="60858d35-14aa-4196-aa0a-d6a9a3b04072" src="/sites/default/files/inline-images/Image%202.png" /></p> <h3 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Near Real Time:</span></h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">MuleSoft provided Connector can be scheduled to bring in data which makes it near real-time depending on the scheduler settings. Connector can be used to Query the Salesforce objects very much like SQL queries.</span></p> <h3 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Event Driven:</span></h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Internal to Salesforce, we can define to trigger an event with defined criteria. When the action happens triggering the event, the data can be sent to integration using the integration URL. In addition to sending data to URL, Salesforce has the capability to store the data in Queue before sending it out to remote systems. This gives a much-needed reliability for the data. Queues can be configured to retry x number of times with x interval. Depending on the use case Queue can be set up for reliability either in Salesforce or an integration layer.</span></p> <h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Inbound  (Integrate Remote Systems to Salesforce):</span></h2> <h3 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Real Time:</span></h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">MuleSoft does provide a very robust Connector for the Salesforce to integrate which can be used for real-time updates to Salesforce.</span></p> <p><img alt="Salesforce Connector" data-entity-type="file" data-entity-uuid="ba0db125-5078-45e3-8977-26aa8904a475" src="/sites/default/files/inline-images/Image%204.png" /></p> <h3 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Bulk Data Transfer:</span></h3> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">The connector in itself provides features for the bulk data transfers. However, with bulk operations, there are chances that integration could overload Salesforce. All the bulk data transfer use case should be thoroughly tested for resource utilization.</span></p> <p><img alt="Salesforce Connector Confiuragion" data-entity-type="file" data-entity-uuid="744a5cca-2bda-43cd-9e30-b2ddd77bff20" src="/sites/default/files/inline-images/Image%205.png" /></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">There is no silver bullet on the pattern to be used, it all depends on the use case.</span></p> </div> </div> </div> </div> </div><div class="container comment-container"> <div class="text-align-center"><h2>Join the Conversation</h2> </div> <div class="row"> <div class="col-sm-12"> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=724&amp;2=comment_node_blog&amp;3=comment_node_blog" token="-oLlOb5u0r_e8a_gw_dCRuRYOgjHGfdq1AhTu5yqDnU"></drupal-render-placeholder> </div> </div> </div> <div class="container"> <div class="row"> <div class="col-sm-12 blog-tags"> <div> <div><a href="/blog%3Fbid%3D332" hreflang="en">MuleSoft</a></div> </div> </div> </div> </div> <div><drupal-render-placeholder callback="Drupal\block\BlockViewBuilder::lazyBuilder" arguments="0=views_block__blogs_block_4&amp;1=default&amp;2=en" token="jBjf0n8iki9sjPwElNYA4B92vVETYVFMv7O9pm9FdHQ"></drupal-render-placeholder></div> <div> <div>About the Author</div> <div></div> </div> </div> </div> Thu, 13 Sep 2018 19:08:35 +0000 Karthik Dega 724 at https://www.avioconsulting.com https://www.avioconsulting.com/blog/how-integrate-salesforce-mulesoft#comments MuleSoft Anypoint Visualizer https://www.avioconsulting.com/blog/mulesoft-anypoint-visualizer <div class="avio-content"> <div class="page-title"> <div class="lead-image"> <div> <div>Blog Lead Image</div> <div> <img src="/sites/default/files/default_images/network-pen-blog-3_0.jpg" width="2000" height="1333" alt="Avio Consulting" typeof="foaf:Image" /> </div> </div> <div class="dark-overlay"></div> </div> <div class="text-overlay"> <div class="container"> <div class="row"> <div class="col-sm-12"> <div><h2> <a href="/blog/mulesoft-anypoint-visualizer" hreflang="en">MuleSoft Anypoint Visualizer</a> </h2> </div> </div> </div> </div> </div> </div> <div class="main-content"> <div class="container"> <div class="row"> <div class="col-sm-12 node-body"> <div> <div><p style="margin:0in 0in 8pt"> </p> <img alt="visualizer" data-entity-type="file" data-entity-uuid="7ce6edfb-fb31-438e-a278-ebbb5d85e6d4" height="62" src="/sites/default/files/inline-images/visualizer.png" width="200" class="align-center" /><p style="margin:10pt 0pt 0pt"><span style="line-height:200%">At the end of June, MuleSoft released an Anypoint Platform tool that I am so excited about, called the Anypoint Visualizer. As the name implies, the Visualizer provides visibility into your application network, giving you additional insight into your APIs, including:</span></p> <ul><li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Which layer an API resides within (i.e. experience, process, system)?</span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Who is using the API?</span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">What are the dependencies of an API (other APIs, databases, backend systems, etc.)?</span></li> </ul><p style="margin:10pt 0pt 0pt"><span style="line-height:200%">As you build out your application network, the above details become really important, providing a better understanding of your runtime, as-built architecture. This is needed, for example, to understand dependency requirements during impact analysis; to enforce governance of layers – ensuring APIs are used correctly, or to use as a runtime dependency tool for troubleshooting purposes. </span></p> <p style="margin:10pt 0pt 0pt"><span style="line-height:200%">Typically, on a project, I would keep track of these details in various sites including Confluence and Exchange</span></p> <ul><li style="margin: 1pt 0pt 0pt;"><span style="line-height:100%">Listing of the APIs a process API invokes</span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Documenting which databases a system API invokes</span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Which API a proxy API is associated with</span></li> </ul><p style="margin:1pt 0pt 0pt"><span style="line-height:200%">While helpful, it was still only consisted of static data, which is why I was ecstatic to see this type of tool now available within Anypoint Platform. With the Visualizer, the gap is now filled, documenting these inter-dependencies in real time. </span></p> <h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Features of Anypoint Visualizer</span></h2> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">To enable an API to be displayed within the tool, there are a just few pre-requisites (see Caveats section for details). The good news is that there are no updates required to code, just simply deploy the API. After it is deployed it will show up within the Visualizer, allowing you to categorize it into the API level:</span></p> <img alt="Mule Visualizer Features" data-entity-type="file" data-entity-uuid="9c09e788-22c4-4265-a78b-6e7ecb24593d" height="274" src="/sites/default/files/inline-images/image%201.png" width="466" class="align-center" /><p style="margin:1pt 0pt 0pt"><span style="line-height:200%">To understand the inter-dependencies, invoke the API. Connectivity lines will then be drawn between the inbound application, APIs, and backend systems, etc.</span></p> <h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Layers</span></h2> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Experience, Process, and System layers are provided OOB, but the outer layer names can be customized to denote, for example, the consumer layer or back-end system layer:</span></p> <img alt="Mule Visualizer Layers" data-entity-type="file" data-entity-uuid="a265329d-86d4-4b1b-b0aa-0be95637c85c" height="408" src="/sites/default/files/inline-images/image%202_0.png" width="852" class="align-center" /><h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Application Identification</span></h2> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Understanding who is using your API can be facilitated by applying the Client Id Enforcement within the API Manager. Within the Anypoint Visualizer, this attached policy will provide a visual depiction of the API consumers. In the above screenshot, you probably noticed the consumer layer has a couple of inbound connections: External Traffic and MyTestApp. External Traffic is the default name for APIs not protected by Client Id Enforcement. But for those that do have this enforcement, the Visualizer will denote the client id of the requesting application (renamed to MyTestApp in above screenshot). This allows you to understand who is using the API, along with how they are using it.</span></p> <h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Metrics</span></h2> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">CPU utilization along with memory usage is also available in the Anypoint Visualizer screen, along with a click into the link to the dashboard for a given API:</span></p> <img alt="Mule Visualizer Metrics" data-entity-type="file" data-entity-uuid="7c363fc0-c206-4470-912e-da94678574bc" height="413" src="/sites/default/files/inline-images/Image%203_0.png" width="254" class="align-center" /><h2 style="margin:20pt 0pt 0pt"><span style="line-height:107%">Caveats</span></h2> <p style="margin:1pt 0pt 0pt"><span style="line-height:200%">As with most first releases, there are caveats that need to be considered:</span></p> <p style="margin:1pt 0pt 0pt"><span style="line-height:100%">Only keeps history for 7 days</span></p> <ul><li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Works with Mule Runtime version 3.9.1 and 4.1.3 (need to redeploy with new runtime)</span></li> <li style="margin:5pt 0pt 0pt"><span style="line-height:100%">Only works for APIs deployed to CloudHub, not Mule on-premise</span></li> </ul><p style="margin:1pt 0pt 0pt"><span style="line-height:200%">Check out this new tool in Anypoint Platform!</span></p> </div> </div> </div> </div> </div><div class="container comment-container"> <div class="text-align-center"><h2>Join the Conversation</h2> </div> <div class="row"> <div class="col-sm-12"> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=721&amp;2=comment_node_blog&amp;3=comment_node_blog" token="zf0YPgTaK9juPB2lsvxY3w-bnwFLxQT9TWVLQ438bXI"></drupal-render-placeholder> </div> </div> </div> <div class="container"> <div class="row"> <div class="col-sm-12 blog-tags"> <div> <div><a href="/blog%3Fbid%3D332" hreflang="en">MuleSoft</a></div> </div> </div> </div> </div> <div><drupal-render-placeholder callback="Drupal\block\BlockViewBuilder::lazyBuilder" arguments="0=views_block__blogs_block_4&amp;1=default&amp;2=en" token="jBjf0n8iki9sjPwElNYA4B92vVETYVFMv7O9pm9FdHQ"></drupal-render-placeholder></div> <div> <div>About the Author</div> <div></div> </div> </div> </div> Sun, 15 Jul 2018 20:49:31 +0000 Jennie DeRosa 721 at https://www.avioconsulting.com https://www.avioconsulting.com/blog/mulesoft-anypoint-visualizer#comments