Trends Shaping “Post-Pandemic” Enterprise Integration – A Practitioner’s View

The pandemic has wired global enterprises to become thriftier and value-focused with a view to get their technological ecosystem more efficient, lean, and meaningful, which can cater to the larger strategic goal of being effective in addressing user and customer experience. Basically, the world has moved notches towards being an “Experience Economy” than a “Feature Driven Economy.” 

This movement is being felt in every stratum of an enterprise’s architecture, including the Integration layer. With more and more enterprises migrating their workloads to the cloud, not just one, but multiple of them, and with each of the cloud “ecosystems” having their own native integration solutions (Azure Integration Solutions, Amazon API Gateway, GCP Integration, SAP PI/PO, etc.), it becomes imperative for enterprises to not only utilize the native integrations for enhanced service efficiency but to also have an Enterprise Integration Platform(EiPaaS) in the center that can be the conduit for cross-ecosystem interoperability.

Likewise, as the strategic prerogatives tend more and more towards “experience,” reusability and scalability of existing APIs and integrations assume greater importance. This apart, the ability of an enterprise to create a Center for Enablement (C4E) that can be business and domain-focused, as opposed to being technology-focused (as was the situation earlier), is gaining a lot of currency in these times. The whole idea is to keep things simple. If any API is working in a certain functional area, the same should be made available to other similar functional areas across geos to drive larger RoI for the enterprises. 

The third trend being observed is the ability to bake in native “observability” into the Integrations so that real-time operational monitoring and reporting can take place at the level of integrations themselves. This will entail capturing key metrics around API requests, API status, Integration Compliance, API latency, etc., and the ability to draw thresholds to get proactive alerts that can enable the Site Reliability Engineers to take pre-emptive measures towards service continuance and resiliency.

Figure 1: Emerging trends in the Enterprise Integration space

These trends capture the spirit of co-existence (use the best-of-breed solutions), the intention to scale and replicate than build and build more, and finally, the ability to track, measure, alert, and act based on insights derived from native observability functionalities. Thus efficiency, scalability, and observability are the three key decision markers for enterprises today while evaluating Integration solutions.

Let’s look at each of these three trends in detail.

1. Co-Existence of Ecosystem Integration With EiPaaS

In a multi-cloud world, where enterprises are parking their workloads in multiple cloud environments while maintaining their most-essential, confidential, and risky workloads on-premises, it is a scenario that calls for various types of integrations – between on-premises and cloud, between cloud and on-premises, between cloud and cloud and between applications on-premises. Several components, applications, and systems need to interact across these environments to ensure seamless data integration, due process orchestration, and rightful end-user experience. Often, the decision-makers in an enterprise become spoiled for choice and hence confused about what should be the “go to” Integration solution that can orchestrate value across the many environments.

Well, there isn’t any silver bullet solution to this quandary. But what should guide the enterprises is using the “best of the breed” approach where they leverage the native integrations of a given ecosystem (Azure Integration Solutions, Amazon API Gateway, GCP Integration, SAP PI/PO, etc.) for integrating all applications located in that ecosystem. Therefore, all applications stored and run out of the Azure ecosystem can leverage Azure APIs and Azure Integrations. Likewise, clients leveraging SAP or Oracle in a heavy way for their ERP needs will do well to leverage SAP PI/PO or Oracle Cloud Integrations for their respective applications. This will not only drive better runtime efficiency but also allow enterprises to derive maximum value from that ecosystem.

The above solves one side of the problem. So, each of our ecosystem applications is well integrated within that ecosystem by using the native integration functionalities of that ecosystem. Within those ecosystem siloes, we have well-oiled, perfectly synced machinery. But come out of these ecosystem bubbles, and you may see distinct, siloed, nonintegrated ecosystems that are struggling to interact with each other. While most ecosystem platforms have “connectors” these days to connect to another ecosystem but imagine a customer with 10-12 different ecosystems within the enterprise, and there could be 120+ connector interactions outside of these ecosystems that would make the enterprise stare at a spaghetti set up, drawing them back to the same problem that they want to get out of.

Enter Enterprise Integration Platform that becomes a one-stop orchestrator of all cross-ecosystem interactions. All connectors from each ecosystem meet the common EiPaaS hub, which will be capable of reading, translating, converting, and exposing messages in the target ecosystem-friendly language and hence becomes a common point of interaction, as shown below.

Figure 2: From a spaghetti “each connector to another” pattern to an “all to one iPaaS” pattern

The above solves the second problem. Therefore, we have a “best of the breed” approach where an enterprise can use all “native” integration capabilities that the ecosystem provides. In contrast, we have an EiPaaS Integration Hub, which connects the dots between the ecosystems.

Below is an illustration that shows how “MuleSoft,” a leading Enterprise Integration platform, becomes a central integrating hub that brings together all ecosystems and on-premises systems. In contrast, the ecosystems themselves integrate the applications in them using their native functionalities.

Figure 3: Sample: “Co-existence” of iPaaS platform and ecosystem integration platforms

Enterprises should respect each platform for its own benefits, leverage native integrations for the application ecosystem, and leverage enterprise integration tools for integrations outside the application ecosystem. The technologies shown in the figure above are for illustration only.

2. Scalability of Integration Best Practices With an “All-Encompassing” C4E (Centre for Enablement)

“All-encompassing” is the operative term here for a reason. Enterprises today use different integration technologies for different use cases. Different units and functions use various integration technologies too, as per their need, expertise, and comfort factor. With the empowerment of Citizen Technologists, low code and no code integration tools like Workato, Tray, Zapier, etc. are also sitting in the overall enterprise landscape, alongside more strategic platforms like MuleSoft, Boomi, TIBCO, etc., which also co-exist as seen in the trend above, alongside ecosystem integration functionalities. Thus, suddenly, an enterprise looks at a smorgasbord of technologies, each with its niche requirement and each being used by a separate team or a group of teams!

The challenge is how to govern a wide swathe of technologies and teams using them and, more importantly, how to make the integrations reusable, replicable, and scalable across geos and units for similar use cases. Therein lies the value.

Many questions and options arise. For example, should a “decentralized model” exist where we have one CoE governing each business unit? Or one CoE governing all the work being done using an integration technology? Or should a “centralized model” exist where one common CoE orchestrates and governs the work across all business units or technologies? Or should there be a “hybrid model” where one Hub CoE interacts with a few “Spoke” CoEs in a hub and spoke” model, and the “spoke CoEs” in turn support a group of business units?

There are pros and cons to each of these models. For example, the “decentralized model” is more suited for large enterprises with independent business units. But lack of consistency across the units can breed a shadow IT culture. Likewise, the “centralized model” is more suitable for small and medium enterprises and start-ups which have a small central IT running the show. Finally, the hybrid model is more suited for large enterprises that encourage autonomous business units, which run in a product organization model and want to be self-sufficient with a centralized, horizontal CoE for any governance and utilities support.

While each of these options is germane, one factor that can truly accommodate for all these options to co-exist is the provisioning of an “all-encompassing” C4E (Center for Enablement) that is truly “extended” and “accommodates” all teams, technologies, and practices under its ambit, governs them, supplies the enterprises with common standards of working, best practices enabling business divisions to build and drive the consumption of assets successfully, enabling speed and agility. It allows the business and IT teams to shift from a production-based to a production-and-consumption-based delivery model. This production and consumption model fuels reusability and scalability.

The C4E is a broader concept than CoE. While the CoEs develop, run, and manage APIs, their operational elements and are a container of functional expertise in a centralized manner, C4Es are enablers, capability builders, decentralized mechanisms of governance that bring in best practices, reusable templates, evangelism, and community building. In addition, they are domain focused and make the business units self-sufficient, independent, and scalable.

3. Observability Baked Into Integrations for Reactive Knowhow and Proactive Alerts

Observability is a comprehensive set of capabilities around logging, analytics, monitoring, troubleshooting, measuring platforms, runtimes, and applications. Observability plays a decisive role in providing the real-time performance of a system. Lacking observability in a system would be like flying an aircraft without a fuel indicator.

Observability is not only shifting left but also diving deep into each layer of the enterprise architecture. From the systems to applications to the processes to the integrations between them, enterprises expect observability and monitoring mechanisms to be baked into each of these elements so that not only is the performance reported but with the application of AI and by drawing clear performance thresholds, proactive alerts can be sounded out to the SREs, thus ensuring service resiliency and continuance. 

In this regard, most Integration platforms today come laced with out-of-the-box (OOB) support for many of the key observables. However, what can be observed and monitored also depends on the deployment option- Cloud, On-Prem, or Hybrid.

Some of the key observables around Integration that the OOB functionalities can observe and report are:

Figure 4: Key Observables in an Integration Solution

There are various third-party observability tools in the market that fill the vacuum left by the OOB functionalities of the Integration platforms. For example, Splunk, ELK, and SumoLogic are commonly used for log aggregation and analysis, while the likes of Jaeger, Prometheus, Datadog, etc., are popular for their tracing capabilities.

However, tools alone do not solve the problem of observability. Having an experienced team of reliability engineers who can interpret metrics and act proactively will be key to success. As they say, it is not data but the action taken on the insights derived from the data which determines true success. Therefore, partnering with the right service partner will be key to Enterprise Integration success.

Conclusion

The Enterprise Integration market is in flux post-pandemic as effectiveness, efficiency, cost optimization, and performance precision are all expected by enterprises in their Integration solution. These are seemingly contrarian demands, but a balanced solution that can ensure the co-existence of diverse solutions for diverse use cases, with an ability to scale and a capability to observe and report key metrics, is key to enterprise integration success. In addition, while the technologies are available in the wider market, a capable service partner with experience in deploying, managing, orchestrating, scaling, and bringing about governance support will be key to Integration strategy success.