CIOs across enterprises run various applications that generates, and transforms loads of data. The data thus generated cannot remain silo. Applications need to be integrated for passing data across to allow collaboration among cross functional teams. There are various designs that industry has devised over time to integrate such applications. One such popular architecture style is Hub and Spoke. Hub and Spoke architecture integrates various applications through a centralized hub.
Many patterns evolved over time to implement this design and most of them are popularly followed. With this article, I’d like to throw some light on anti patterns to raise awareness. These are some strict “Don’ts“ while integrating applications using a centralized hub.
Using as surrogate to existing data stores: Applications and databases in enterprise serve different purposes. Individual databases are setup for handling operational or analytical data. Integration Hub(IH) should not try to emulate such existing applications or data stores. Main purpose of IH should remain for just flowing data to destination applications. IH should not own spoke's responsibilities which would require domain expertise.Forming tight coupling with source or destination systems : One of the main objective in enterprise applications landscape is to develop loosely coupled systems. It could be connections, data formatting, or protocols development involved in integration. Each area should be independent, modular and loosely coupled. Loosely coupled systems are easy to scale, remain flexible, and interoperable. Adopting event driven architecture is one best way to makes systems loosely coupled. These advantages in turn bring monetary benefits.
Hard-coded integration : Integration logic or configurations hard coded in the systems might provide near-term conveniences. The configuration embedding in the code approach seem easier and faster, but it end up creating a rigid and inflexible integration that is difficult to change, or reuse. For example, if the integration requirements or rules change, the code has to be modified and redeployed, which can introduce errors and downtime. To avoid this anti-pattern, use an externalized and declarative integration approach that can separate the integration logic and configuration from the code, and store them in a configuration file or a database.
Insufficient Data Validation : Not validating passing data at earlier stage, could build up data sync issues in multiple systems eventually. Even with event driven architecture, event schemas that lack validation makes integration systems more vulnerable to errors surfaced during development phase. Most often, application to be integrated are turn out to be heterogeneous.
Not Using Event/Message Replay : More often, receiving system requires data from past to catch up due to unintended delays. Replay feature being unavailable, leads to humongous operational effort to republish data from certain point.
Lacking data lineage mechanism: Data lineage provides detailed map of how data is ingested, transformed, and activated across the data pipeline. Without that, root cause analysis during incident resolution becomes harder. A complete data lineage implementation should include details about data sources, data transformations, data destinations, metadata, and dependencies between different data elements. This mechanism not only helps in tracing but also improves data quality.
Not Monitoring for key metrics : Integration systems should be planned to measure metrics in real time. It helps in measuring data volumes and assess trends over time. Sudden hikes and drops in volumes usually are symptoms of failed components. Detecting anomalies with proactive monitoring, helps save cost and reputation.
Poor Error Handling: No system can run error free. Every system should have fail safe design with capable error handling mechanism. It handles failure that can be caught and retried. When building assured delivery systems, not a single event or transaction are allowed to be ignored. Accurately pointing the error sources and preempting is absolute necessary. Event retries should become part of error handling component to ensure data is delivered to destination. System should adopt plans for short term, long term retries and also manual intervention when necessary.
Ignoring Scalability: Poor scalable integration systems negatively impact metrics like data volume, sync time, and data accuracy. Such poor performance affects business continuity. Integration hub components should have capacity planning to handle varying data volume, and resiliency to disaster recovery. Replicas running in different availability zones or data centers should be considered to handle infra failures.
Missing self serve provision : Centralized integration hub can empower businesses to establish a unified view of their data, breaking down data silos and promoting cross-functional data sharing and collaboration. Cross functional teams should own and be able to integrate systems without steep learning curve. Integration Platform should be developed in a way to allow this.
Delegating data security and governance responsibilities to Spokes: Although integration hub keeps data only in transit, it is still paramount to keep data in motion secure and governed for meeting compliance requirements. Features like access control, standard encryption, and confidential data handling, data masking and etc. should be well supported.
The above list serves only the most pressing anti patterns. I'm sure there could be many other ways that provide clarity on what "not to do" while developing integration hub. With ever changing marketing dynamics and technology landscape, technologists will be uncovering them to constantly improve the integration metrics.
Hope you find this article useful.
