WSO2 is now Choreo LabsLearn more

Trusted AI Governance
Build

Create agents and AI apps with the tools and languages your teams already use.

Connect

Link agents securely to your APIs, data, models and business systems.

Run

Deploy, observe and operate agents and AI apps reliably at any scale.

Govern

Control identity, access, cost and policy for every agent in production.

Agent Management

Developer Platform

  • API Management Cloud

    Design, secure and publish APIs with governance and analytics built in.

    Sign Up Now
  • Integration Cloud

    Connect apps, data and AI agents with low-code and pro-code integration.

    Sign Up Now
  • Identity & Access Management Cloud

    Secure sign-in and access for customers, partners, workforce and agents.

    Sign Up Now
  • Agent Management Cloud

    Build, run and govern AI agents with full visibility and control.

    Sign Up Now
  • Developer Platform Cloud

    Ship cloud-native apps faster on an internal developer platform.

    Sign Up Now

Self-Hosted

Full control over your stack. Deploy directly to your own servers, bare metal, cloud, or Kubernetes environment. Your data never leaves your perimeter.

Product Downloads
Choreo Labs
AI Governance
Platform
Solutions
Resources
Article

[Blog post] entity aggregation with WSO2 enterprise service bus and WSO2 data services server – part I

The integration problem

Businesses continue to grow and as they expand over the years the demand for IT systems also increase. Mergers and acquisitions became the standard for growing businesses and the demands for new IT systems skyrocketed. As a result, today's enterprises are composed of multiple IT systems that are independent of each other. The result is hundreds, if not thousands, of application islands. For the business to sail smoothly, all the data in these isolated applications must be easily accessible to each other. This usually creates an integration nightmare.

What are these particular Integration problems?

  • Organizations find themselves struggling with significant problems of data integration among a wide variety of data sources.
  • The mismatch between the data representation that one application provides and the data representation that another application expects forms the core of this long-lived problem.
  • This data integration problem is particularly troublesome for SOA implementations because it limits the loose coupling between service providers and consumers.

Integration needs can fall into these categories:

  • Standard Integration – This is the straightforward need for integrating applications and systems for data sharing.
  • Entity Aggregation – This is enforcing consistency between systems sharing same entity information by maintaining a holistic view that is concerned with the whole instead of just the parts.
  • Standards Compliance - Adherence to standards insures the loose coupling and composition of services and data exchange mechanisms with a guarantee that the data exchanged with an individual service will be universally comprehensible e.g. HL7 in healthcare information systems.

What is entity aggregation?

There are essentially three reasons for the aggregation of entities in different systems:

  • Single View of Entity—having a single view of all common entities spread across enterprise systems. E.g. a unified view of a Customer entity, named Customer Information Repository (CIR), is required to build useful solutions in many enterprises, especially in the banking sector.
  • Horizontal Partitions—there are situations where entity information is distributed across many different services, based on factors such as geographical boundaries.
  • Cross-Join scenarios—these situations require cross-join among entities present in different services.

- (Kothandraman, 2004)

Goal of an aggregation service

We can use an aggregation service to embed the transformation logic used to connect the disparate systems and expose a unified view of the entity data. This transformation service, loosely-coupled in a SOA implementation, accepts incoming data in one format, and outputs data with the same semantic meaning in another format, which will be the holistic view that is required from an entity aggregation service. The use of a data transformation service or an aggregation service in this case, improves loose coupling by separating the concerns of service providers and consumers.
Some goals of having an aggregation service:

  • Acts as a unified source of entities.
  • Provides a holistic view of an entity.
  • Provides a holistic view of the entity model—entities and their relationships with other entities.
  • Enriches the combined data model across systems—the whole being better than the sum of its parts.

- (Kothandraman, 2004)

Reference architecture: Entity aggregation

In this particular organization, information of the Customer entity resides in two systems – in their Customer Relationship Management and Enterprise Resource Planning systems. As shown in the diagram, the systems share the same entity (Customer) while the data models are slightly different. Some fields are duplicated while some others are additional information that one system lacks. An Entity Aggregation solution can be designed to not only act as a single point in accessing information that exists in multiple systems, but it also provides a holistic view of an entity and the entity model. The entity aggregation service communicates with the CRM and ERP systems and creates a whole new data model using some transformation logic (in this case it uses XSLT transformation). This holistic view can be exposed as an API to parties interested in retrieving all the information related to a customer in this organization.

WSO2 solution architecture: Entity aggregation

A solution to act as the data aggregation service can be implemented with the WSO2 Enterprise Service Bus or WSO2 Data Services Server or a combination of both depending on the systems that need to be integrated.

If both the CRM and ERP need to be integrated at the application level, the WSO2 ESB can integrate the systems and expose a unified view as a REST API or a proxy service which is a SOAP service. The WSO2 Enterprise Service Bus supports rich transformations using the transformation mediators belonging to its comprehensive mediator library such as XSLT, FastXSLT, XQuery, Enrich and Payloadfactory mediators.

If both the CRM and ERP need to be integrated at the data level, where the information needs to be extracted from two databases and then transformed to a RESTful or SOAP service, the WSO2 Data Services Server can be leveraged. While the Data Services Server can expose data from various data sources – e.g. RDBMS data sources, Excel spreadsheets, CSV, Google spreadsheets etc. – it also supports transformation using XSLT transformation, where the user can define the transformation in XSLT.

If one system needs to be integrated at the application level and another needs to be integrated at the data level, which is the case in the above diagram, the ESB alone or a combination of the ESB and Data Services Server products can be used.

Part II

The implementation details of the above scenario using the WSO2 ESB and WSO2 Data Services Server will be covered in the 2nd part of this blog post.

References

Kothandraman, R. 2004. SOA Challenges: Entity Aggregation, https://msdn.microsoft.com/en-us/library/ms954596.aspx

Click here to refer to Dakshitha's blog.