IBM App Connect Enterprise Pipelines and Integration Nodes

With the current focus across the industry on containerization, it may seem like the older solutions such as integration nodes are almost deprecated, but this impression is wide of the mark and nowhere more so than in pipelines. Modern software development methodologies such as unit testing and continuous integration can apply equally well to integration node solutions, and there is now a public demo to show how this can be achieved.

See sections below: ACE tests run everywhere and the demo pipeline now shows them running followed by integration node deployment.

Background

App Connect Enterprise applications have historically been deployed and run on integration nodes (“brokers”) in datacenters and developer laptops, and many customers have created sophisticated deployment pipelines to ensure only high-quality code is put into production. This normally includes the source being built into BAR files either manually or with automation, and the BAR files then progressing through multiple environments (Dev, QA, Pre-Prod, Prod, etc.) with both manual and automated testing at the various stages to validate the application code and configuration.

While this has proved to be a very effective way to build and run integration solutions supporting the most critical business applications, the industry-wide move towards containers requires some modifications to the picture, as does the modern expectation of continuous integration (CI) and continuous delivery (CD) builds that require code to have automated unit tests. The agile integration architecture approach shows the progression toward containers where the applications in the centrally-managed integration nodes on the left are progressively moved into containers for greater deployment agility (amongst other benefits):

In common with the rest of the industry, there is an assumption of automated testing being built into a CI/CD pipeline when building container images for ACE, as shown in a picture from the ACE demo pipeline:

However, containerized deployment and unit testing are two separate challenges, and are not intrinsically linked: it is quite possible to implement one without the other, and this includes being able to run unit tests in a pipeline that deploys to an integration node.

Note that although this article is focused on unit testing with an integration node as the pipeline endpoint, the opposite is possible (though not recommended): running applications in containers without running automated unit testing earlier in the pipeline. For example, the Cloud Pak for Integration (CP4i) allows the upload of arbitrary BAR files to be used in ACE containers, including BAR files that have not been tested in an automated manner. This is not encouraged, and OpenShift Pipelines would be a good choice for building applications and running tests beforehand.

Destination-Agnostic Pipelines and Testing

ACE was designed to be flexible in both application build and container deployment capabilities, as can be seen in a picture from an early ACE v12 presentation showing the progression from the developer on the left to a deployed application on the right, with a pipeline in the middle:

The pipeline destination on the right-hand side is shown as being either containers or an integration node, with the pipeline in the middle running the same stages regardless.

ACE provides capabilities that enable this mix-and-match approach to test and deploy in several key areas:

Building Unit Tests That Run Anywhere

The developer on the left of the diagram has a small activity box labeled “Unit Test” (along with “Prototype” and others), and the ACE v12 toolkit enables the creation and running of unit tests (including assisted test creation) to allow test-driven development as well as other unit testing (see here for details). This allows finer-grained testing of subflows and individual nodes with a fast feedback cycle to allow quicker application development with higher quality.

The unit testing does not stop there. The pipeline also has a stage labeled “Unit Test," and this can run the same tests. In fact, it can run not only the tests being written by the developer in the picture, but can also run other unit tests created by other developers (or tests written in the past by the same developer) to ensure that previous use cases continue to work and that any conflicts between changes made by multiple developers are caught as early as possible and fixed quickly.

Once the tests and applications have been placed in a server work directory, running unit tests can be as simple as:

IntegrationServer -w /tmp/work-dir –test-project Application_UnitTest

This can be run from anywhere, including on-prem virtual machines, containers, developer laptops, etc. The toolkit runs the server this way (with a few more options!) when running JUnit tests, and the ACE server starts up, runs the tests, reports successes and failures, and then shuts down again with an exit code reflecting pass or fail.

Same ACE Server Regardless of Where it Runs

The IntegrationServer used to run ACE flows in containers and unit tests is the same code as the DataFlowEngine in an integration node: the command line to run the two is different, but the flow code is the same. This makes it safe to test a flow in an IntegrationServer in a pipeline and then run the same code in an integration node, as the flow will behave in the same way (assuming the same credentials, policies, etc). Migration to containers at a later date is also safe, as once again the flow code will be the same.

Simpler and Faster Build Tools

Automated BAR builds have been possible for a long time using mqsicreatebar or mqsipackagebar, but the latter is limited as it will not compile some projects while the former requires a lot of infrastructure (see this GitHub repo) and is quite slow. ACE v12 improves this picture with the ibmint command, which does not require the toolkit to be installed but will compile projects as needed (with some exceptions).

The command below would compile DemoApplication and unpack it into the server work directory without needing to create a temporary Eclipse workspace (as mqsicreatebar does). :

ibmint deploy --input-path <workspace> --output-work-directory <workdir> --project DemoApplication

Multiple projects can be specified, and BAR files created if desired rather than unpacking into the work directory immediately (see the ibmint commands documenation for details). This works for test projects as well as applications, libraries, etc.

Connectivity During Test Runs

Testing in the integration world is usually harder than ordinary application testing simply because of the range of services included in integration solutions: integration is usually connecting systems together, and so there tend to be quite a few connections required. Unit testing is still needed, of course, as integration solutions are similar to other applications in that they contain code that must be tested.

There are many schools of thought around allowing unit tests to connect to outside services, ranging from the “purist” view, where unit tests must run without calling services and use mocks instead, to what might be called the “anything goes” view, where tests can do whatever the writer wants. The former ensures that tests can be run anywhere without needing connectivity to critical systems (so developers can always run the tests on their laptops), but lacks the end-to-end coverage provided by actually connecting to a real system.

ACE v12 allows both, and in fact, ACE development runs both “purist” unit tests and also “component tests” in the internal product development pipeline, where component tests are unit tests that are allowed to connect to external services. This distinction makes it obvious to any developer looking at a test (or at the code) whether or not connectivity might be needed: if it’s a “unit test” then it should just run anywhere without any setup, but if it’s a “component test” then it probably needs some dependent service (MQ, a database, CICS, etc). The name “component test” is unfortunately over-used in the software world, but does at least provide clarity if used consistently. See also ACE unit and component tests for further discussion of the difference between test styles in integration.

Standard ACE JUnit tests can be either unit tests or component tests, with the main difference being whether they connect to external systems. For component tests, the server work directory is usually created in advance and policies/credentials are provided, at which point the tests can be run the same way as unit tests.

The next section describes an example with both styles of test in operation in the same pipeline.

Example of Unit Testing With Integration Node Deployment

The ACE demo pipeline mentioned earlier was originally constructed to show an example of both testing and container deployment (including Kubernetes, OpenShift, and CP4i variants) with unit tests for the code and component tests for database connectivity, but has recently been enhanced by the addition of a Jenkins build pipeline that deploys to an integration node:

Full details and instructions are in the repo itself, but the Jenkins pipeline is intended to mirror the existing Kubernetes Tekton pipeline:

• Jenkins builds can be started whenever the source repo has changed (and could be automatically started with a webhook).
• The source is extracted and the application is built from it.
• Unit tests are run, and any failure stops the pipeline.
• Component tests are run using a test database, and any failure stops the pipeline.
• The application is packaged and deployed to a previously-created integration node.

The integration node at the end of the pipeline represents the “Dev” environment, and normally there would be other environments after that. More testing, and usually a manual approval step, would be needed before the application is pushed into production, but it is likely that most organizations already have procedures in place for that aspect. This demo is intended to show how to incorporate ACE unit testing and builds into standard pipeline tools (Jenkins, Tekton, etc.), and does not extend into the later environments.

Summary

Although it may seem that containers and unit testing are inextricably linked and that it’s not worth looking at unit testing until the move to containers is underway, ACE v12 is actually flexible enough to allow progress to be made in both areas independently. Smaller subsections of flows and code can be tested quickly on developer boxes and in pipelines that end in integration nodes, as can larger end-to-end scenarios using the same JUnit technology, without waiting for a larger organization-wide move to containers.