Anti Patterns for Data Integration Hub

CIOs across enterprises run various applications that generates, and transforms loads of data. The data thus generated cannot remain silo. Applications need to be integrated for passing data across to allow collaboration among cross functional teams. There are various designs that industry has devised over time to integrate such applications. One such popular architecture style is Hub and Spoke. Hub and Spoke architecture integrates various applications through a centralized hub.  

Many patterns evolved over time to implement this design and most of them are popularly followed. With this article, I’d like to throw some light on anti patterns to raise awareness. These are some strict “Don’ts“ while integrating applications using a centralized hub. 

Using as surrogate to existing data stores: Applications and databases in enterprise serve different purposes. Individual databases are setup for handling operational or analytical data. Integration Hub(IH) should not try to emulate such existing applications or data stores. Main purpose of IH should remain for just flowing data to destination applications. IH should not own spoke's responsibilities which would require domain expertise. 

Forming tight coupling with source or destination systems : One of the main objective in enterprise applications  landscape is to develop loosely coupled systems. It could be connections, data formatting, or protocols development involved in integration. Each area should be independent, modular and loosely coupled. Loosely coupled systems are easy to scale, remain flexible, and interoperable. Adopting event driven architecture is one best way to makes systems loosely coupled. These advantages in turn bring monetary benefits. 

Hard-coded integration : Integration logic or configurations hard coded in the systems might provide near-term conveniences. The configuration embedding in the code approach seem easier and faster, but it end up creating a rigid and inflexible integration that is difficult to change, or reuse. For example, if the integration requirements or rules change, the code has to be modified and redeployed, which can introduce errors and downtime. To avoid this anti-pattern, use an externalized and declarative integration approach that can separate the integration logic and configuration from the code, and store them in a configuration file or a database.

Insufficient Data Validation : Not validating passing data at earlier stage, could build up data sync issues in multiple systems eventually. Even with event driven architecture, event schemas that lack validation makes integration systems more vulnerable to errors surfaced during development phase. Most often, application to be integrated are turn out to be heterogeneous. 

Not Using Event/Message Replay : More often, receiving system requires data from past to catch up due to unintended delays. Replay feature being unavailable, leads to humongous operational effort to republish data from certain point. 

Lacking data lineage mechanism: Data lineage provides detailed map of how data is ingested, transformed, and activated across the data pipeline. Without that, root cause analysis during incident resolution becomes harder. A complete data lineage implementation should include details about data sources, data transformations, data destinations, metadata, and dependencies between different data elements. This mechanism not only helps in tracing but also improves data quality. 

Not Monitoring for key metrics : Integration systems should be planned to measure metrics in real time. It helps in measuring data volumes and assess trends over time. Sudden hikes and drops in volumes usually are symptoms of failed components.  Detecting anomalies with proactive monitoring, helps save cost and reputation. 

Poor Error Handling: No system can run error free. Every system should have fail safe design with capable error handling mechanism. It handles failure that can be caught and retried. When building assured delivery systems, not a single event or transaction are allowed to be ignored. Accurately pointing the error sources and preempting is absolute necessary. Event retries should become part of error handling component to ensure data is delivered to destination. System should adopt plans for short term, long term retries and also manual intervention when necessary.   

Ignoring Scalability: Poor scalable integration systems negatively impact metrics like data volume, sync time, and data accuracy. Such poor performance affects business continuity. Integration hub components should have capacity planning to handle varying data volume, and resiliency to disaster recovery. Replicas running in different availability zones or data centers should be considered to handle infra failures. 

Missing self serve provision : Centralized integration hub can empower businesses to establish a unified view of their data, breaking down data silos and promoting cross-functional data sharing and collaboration. Cross functional teams should own and be able to integrate systems without steep learning curve. Integration Platform should be developed in a way to allow this. 

Delegating data security and governance responsibilities to Spokes: Although integration hub keeps data only in transit, it is still paramount to keep data in motion secure and governed for meeting compliance requirements. Features like access control, standard encryption, and confidential data handling, data masking and etc. should be well supported. 

The above list serves only the most pressing anti patterns. I'm sure there could be many other ways that provide clarity on what "not to do" while developing integration hub. With ever changing marketing dynamics and technology landscape, technologists will be uncovering them to constantly improve the integration metrics. 

Hope you find this article useful. 

AI Powered API Mocker - Powering Test Automation

Introduction

APIs(Application Programming Interface) are the most popular way to programmatically integrate different applications. Applications in enterprise often required to be integrated with various applications to acquire enterprise data. Most common integration mechanism used is REST API. This also brings challenges in testing the application when it is connected to many other applications.   Most of the time test is conducted to establish connection, validate the contract, checking if API exists as per the contract. Less often it is used to test the actual data in automation space. 

Automation Challenges

The main challenges in testing such applications are: 

• With commercial and SaaS based API servers metering their API consumption, it incurs high cost to consume APIs for testing especially in load testing scenarios.  

• No control on external applications and downtime. 

• No control on network. Network issues block the connection.

• Time consuming external calls

Testing of a system that has various integrations through APIs to other systems, is cumbersome due to the above listed challenges. Such system cannot be tested in silo also. The lack of good integration testing can lead serious failures in production. 

Solution

This solution addresses the above mentioned challenges by developing a system that simulates external API Server to serve the API request but with closely matching data. Data from this system is not reliable and cannot be used for testing. However, this system helps in testing successfully the connections, input validation,  interface contract validation, response format check, and response validation.

 

Benefits:

• As this system is setup locally, it provides reliable network.
• Eliminates API metering cost
• Quicker with option choosing faster network for deployment
• Downtimes can be controlled

 

This simulator system is built as server that receives requests from the application that is subjected for API integration testing. 

Fig-1 : Simple Flow

Mock system is built as server to intercept the outgoing traffic from the application that is subjected for API integration testing. 

Mock system components:

1. Frontend Server:  Receives requests from the application and redirects to ML model serving component.
2. Data loader: Helps in gathering API data from various sources to feed to model for training.
3. Random data generator component
4. ML Model Trainer: To train with API requests and responses to predict accurate responses. Train data for this model is HTTP request parameters, and HTTP reposes parameters which includes both valid and invalid request data.
5. ML model serving component: This serves the outcome of the ML model and send the output to Response Data Builder. 
6. Mock Response Data Builder: Builds HTTP response data in JSON format to send it back to application. 
7. JSON Extractor: Extracts data values from JSON and converts to CSV file. 
8. Data store: To store ML model and input data

    Fig-2 : High Level Solution

Mock Response Builder

This is the core part of the system and here are the approaches adopted to solution the data generation. Adding more details here to make you understand why this is challenging. 

API request and response comes with various types of data. This not just includes datatypes like String/text, Clob/large text. It has to support boolean, chars, numerals(int, long, double), alphanumeric, binary (for images, & files) and etc. API mocker should mock all these types of data in order to successfully fulfil the incoming request like the original server. Every data type poses different type of challenges to the mock system to generate it successfully. A simple random generator may not work well for all data types if results need to match the original server response. 

 

1. Random Data Generator: 

For most of the data types like boolean, numeric, chars, list(repeater), date, country, city etc. random generator solutioning is used. Randomness is a simple way to achieve content generation required in providing as part of response to the API request. Generating object identifiers (IDs) also done using random generator. 

2. Machine Learning for Data Generation:

Machine learning plays crucial role in the solutioning. Response content generation based on the input request is powered by machine learning.  Reading through the below solution overview is a must to understand the role of AI in this solution. 

Evaluated models: Two parametric algorithms are tried in my short project.  A set of labeled examples, with each example comprising of a set of feature values and a corresponding class label is used for training. Model uses it and learn a general rule to classify new examples that are presented to it later. This system is trained with data on external APIs mainly with text data. The scope doesnot cover image and file data types. 

1. k-Nearest Neighbor (KNN)

2. Support Vector Machines (SVM)

SVM showed slightly better accuracy on the dataset I used for one small example. The data set contained text retrieved from mere 100 API request labels. I have not extensively tried with model tuning as the main focus was on building end to end API Mocker without much effort on AI.   

Im planning to try out open-source pretrained(foundational) generative model in future as the trend shows more promising outcomes and removes the hassles of training. 

Model Training

Below are the steps involved in model training:

1. Data Sourcing: Crawling target system APIs, Swagger like API documentation, List of APIs and input params in CSV 
2. Pre processing and Data preparation:
1. Data collected is fed to CSV file format for each API request method. Seperate csv files are created for each API request
2. Data is classified as input parameter and target parameters
3. Data types used : Valid input values, User invalid input values, Boundary conditions
4. Training method: 
1. Training with actual request parameters and responses
5. HTTP Methods to be supported : GET, POST, PUT and DELETE

Fig-3 : Training the model

Real time serving using model output

Configuration specifies the attributes required for data generation through model. When there is a need for text generation is identified, trained model is invoked. Data extracted from request JSON goes through validation phase. Data validation is done as per the trained data(from Swagger API) or manually fed documentation. 

Fig-4 : Real time model serving

Response JSON generation

1. By taking the output of the above model, JSON is constructed. 
2. Respond back with SON format

Most importantly, this application allows API call to original server in case it needs to be tested against it rather than mock server. There will be a toggle switch provided by adding the URL of the application to bypass the mock server and make direct connection to target API server.

Conclusion

The effectiveness of API testing is greatly increased when tests use realistic data, representative of real-world production conditions. Generating tests from production data must be done with care due to the risk of exposing sensitive data. Without automation, creating real-world useful tests is difficult to achieve at scale because of the high labor cost of combing through mounds of data, determining what is relevant, and cleansing the data of sensitive values.

This system is not meant for accurate testing of data, and should never be used when accuracy and precision of API response has to be verified. This is suitable mainly for interface contract testing when testing against original API server is posing challenges as mentioned above. 

Future of Enterprise Application Integration(EAI) Middleware

There are various types of middleware in software industry. This blog limits the scope to EAI middleware that is widely used in enterprises. This type of  middleware adds integration capability to a functional system that end users interacts with. It offers connection to heterogeneous systems, data transformation, fields mapping, routing logic and sometimes rule engine.  Technical teams also prefer middleware to unload certain heavy operations from main functional systems. 

Typical functionalities of EAI middleware