Dell Boomi in Action

Introduction

In the previous Dell Boomi blog post we learnt how the modern cloud-native iPaaS platforms like Boomi are becoming the integration platform of choice for all sorts of use cases. In this post, we'll take a look at Boomi use cases and best practices from our experience of deploying Boomi.

Dell Boomi Integration Cases

• SAP to SaaS Apps: SAP ERP application hosted behind a firewall can send order details to a SaaS application and get the order details back with Boomi.
• AWS RDS to Azure Apps: AWS RDS based DB needs to fetch product details from another system hosted on Azure and cost details from a highly secure on-premise system.
• JSON Data ETL: Data in JSON format written by an upstream system in AWS S3 bucket can be extracted using S3 API in Boomi, transformed into CSVs, flattened and uploaded to Oracle Enterprise Financial tools like UCM.
• Accounting Data Movt: Extraction of accounting data (e.g. cost center, account, department, location) from Oracle via Oracle Reporting Service. This data can be synced to the PostgreSQL DB and then can be sent to NetSuite using SuiteTalk Web Service.
• SFTP to Azure DB: Account information written in CSV files in SFTP site can be upserted to Azure SQL database and the new records inserted are fetched to be uploaded to the SAP ERP application.

Notifications & Alerting

In case of errors and exceptions occurring in the Boomi process (system failure / HTTP Unauthorized error / connection refused error etc), you can consider below approach.

• Raise a ticket in JIRA via Boomi by with process name, error information, assign to a user and add watchers
• Send an alert in slack channel to all the stakeholders using Slack connector
• Write the error information to an application log file using Disk connector
• Write the error message to exception database using Database connector
• Send email notification to all the stakeholders using Email connector

Boomi Best Practices

During our Boomi deployments for our customers here are some of the best practices

1. To process large volumes of data, use Flow Control with Parallel Processes and Run each document individually which spawns parallel JVMs across nodes in molecules and helps in distributing the load equally in all atom nodes and each JVM can process files sequentially.

1. In case of running one time loads (monthly loads / conversion tasks) where huge volumes of data need to be processed (eg. 300 million transactions) enable “Purge Data Immediately” which deletes all process data and temp files from the atom once processing is completed. If not done, disk space could fill up and can result in atom crash or no available space in disk error.

1. When using custom scripting in Data Process shape to apply any custom logic, always use streams instead of string as string loads the entire message in memory but stream writes the message in temp files in the atom and loads a portion of the data in memory to operate on.

In case, it is needed to read the stream line by line using bufferedReader, avoid using bufferedReader.readLine() as it also loads the entire line to memory and can result in atom crash, rather use bufferedReader.read() which reads character array depending on the length given.

char[] charRead = new char[1024];
while((bufferedReader.read(charRead, 0, 1024))!=-1){
// do something
}

1. In case of creating parent and multiple child processes, use ProcessRoute instead of ProcessCall shape. When using ProcessCall, you deploy only the parent process and not the child processes. So, even if one child process (out of many) changes, you will need to deploy the parent process. When using ProcessRoute shape, you need to deploy parent process, child process and ProcessRoute separately. This gives you flexibility of changing and deploying only the child process you change and not deploying the parent process again and again.

Conclusion

In this post we saw some of the possible integration ideas, alerting strategy and some best practices. We hope this post was useful for you and helped you with your own Dell Boomi issues.