WSO2 is now Choreo LabsLearn more

Trusted AI Governance
Build

Create agents and AI apps with the tools and languages your teams already use.

Connect

Link agents securely to your APIs, data, models and business systems.

Run

Deploy, observe and operate agents and AI apps reliably at any scale.

Govern

Control identity, access, cost and policy for every agent in production.

Agent Management

Developer Platform

  • API Management Cloud

    Design, secure and publish APIs with governance and analytics built in.

    Sign Up Now
  • Integration Cloud

    Connect apps, data and AI agents with low-code and pro-code integration.

    Sign Up Now
  • Identity & Access Management Cloud

    Secure sign-in and access for customers, partners, workforce and agents.

    Sign Up Now
  • Agent Management Cloud

    Build, run and govern AI agents with full visibility and control.

    Sign Up Now
  • Developer Platform Cloud

    Ship cloud-native apps faster on an internal developer platform.

    Sign Up Now

Self-Hosted

Full control over your stack. Deploy directly to your own servers, bare metal, cloud, or Kubernetes environment. Your data never leaves your perimeter.

Product Downloads
Choreo Labs
AI Governance
Platform
Solutions
Resources
Article

Batch processing with WSO2 ESB - an overview

This article demonstrates the batch processing capabilities of WSO2 Enterprise Service Bus (ESB).

Pre-requisite

WSO2 ESB

Introduction

Most legacy applications produce batch files that contain hundreds of entries, which need to be processed by other applications as individual
messages. This brings about a need to introduce an integration layer that's capable of processing these batch data and transforming them to messages that can be
consumed by other applications in the organization’s eco-system. This article looks at how the WSO2 ESB can be used to process batch data and transform
them to individually consumable messages that can be sent to other applications in the organization.

Applies to

WSO2 Enterprise Service Bus

4.5.0 or above

Table of contents

  1. Processing batch data
  2. WSO2 ESB to process batch data
  3. The Scenario
  4. Summary

Processing batch data

A batch file comprises multiple records that need to be processed before they can be sent to other systems in the organization. This process would
require breaking down the content of the batch file into individual messages that can be directly consumed by other systems in the organization. This
brings about a need for message mediation whereby a 'mediation engine' would be required to break the batch data into individually consumable messages and then
transform them to the message formats that are expected by other systems in the message flow. This process can be illustrated by the diagram below.

Batch transformation

WSO2 ESB to process batch data

WSO2 ESB is capable of acting as the mediation engine that can process the batch files to create individual messages that can be transformed to message
standards, which are expected by other systems. WSO2 ESB can process batch files via the VFS (virtual file system) transport, which can listen to a given file
location and pick up files for processing.

The Scenario

The scenario involves a legacy system that produces a CSV file containing customer information that needs to be processed by a CRM application. The CSV
file contains information of multiple customers that needs to be processed individually by the CRM application. It is also required to transform the
information in the CSV file to a message format that is expected by the CRM application. WSO2 ESB would be used to process the batch file, transform the
messages, and send the messages to the CRM application. This scenario can be depicted by the diagram below.

Batch transformation

As illustrated in the diagram above, the CSV files from the legacy system is picked up by the ESB via VFS transport. The content of the file is then
transformed to the XML using a Smooks file. This file will contain all the records in the CSV file as a single XML. WSO2 ESB will then break up individual
messages out of this XML file and build a message payload that would conform to the message standard expected by the CRM application. Once the message
payload is built, individual messages would be sent to the CRM application. The above-mentioned process would transform a CSV file containing the following
content.

0001,3463,3,12.50
0002,3452,2,4.50
0003,3463,1,6.20
0004,5421,5,3.40

Into a series of messages as given below.





0001
3463
3
12.50







0002
3452
2
4.50







0003
3463
1
6.20







0004
5421
5
3.40




Running the scenario

  1. Enable VFS transport on the ESB instance - the VFS transport is disabled in the vanilla pack of the ESB. It can be enabled in the
    <ESB_HOME>/respository/conf/axis2/axis2.xml file. Enable the VFSTransportListener and VFSTransportSender as shown below.
    
    
    
  2. Add the Smooks file that would transform the CSV file to XML - Include the smooks-csv.xml file in the following location
    <ESB_HOME>/repository/samples/resources/smooks/. The file is provided as an attachment to this article.
  3. Start the WSO2 ESB
  4. Add a local entry that stores the location of the Smooks file - the local entry will store the location of the Smooks file that can be used within the
    ESB sequence. Given below is the configuration for the local entry
    
       
    
    
  5. Create a Proxy service - a proxy service needs to be created, and would need to have the mediation logic required to transform a batch file into individual
    messages that can be sent out to the CRM service. Given below is the proxy service configuration
       
          
          
             
                
                   
                   
                
                
                   
                      
                         
                            
                               
                                  $1
                                  $2
                                  $3
                                  $4
                               
                            
                            
                               
                               
                               
                               
                            
                         
                         
                         
                         
                         
                         
                            
                               
    MOVE 5 file:///D:/Demo/BatchFiles/processed/ file:///D:/Demo/BatchFiles/in/ file:///D:/Demo/BatchFiles/failure/ .*.csv text/plain MOVE
  6. Add the CSV file to the folder location provided in the vfs.FileURI path.

Summary

As demonstrated in this article, the WSO2 ESB has the capability of acting as a mediation engine that can process files containing batch data into
individual messages, which can be consumed by other applications in the organization.

Attachment Size
smooks-csv.xml 348 bytes