Exploring IBM App Connect Enterprise 12.0.12.0

We'll explore five of the features released in IBM App Connect Enterprise 12.0.12.0 and provide a high-level overview of each;

1. Using an HTTP Proxy with the Discovery Connector Message Flow Nodes
2. Runtime Monitoring Events Specify File Location
3. Path Tracking in the Webui
4. Improved Java 11 support
5. Blocking the retrieval of credentials stored in a vault

Using an HTTP Proxy With the Discovery Connector Message Flow Nodes

For the last several mod releases, ACE software has been expanded with large numbers of new Discovery Connector message flow nodes which are typically used when integrating with third-party SaaS applications. When deploying message flows containing Discovery Connector nodes within an on-premises data center, for some users it might be considered necessary to go through an HTTP proxy when connecting to such public endpoints. With this requirement in mind, ACE 12.0.12.0 provides four specific Discovery Connector nodes with this capability:

• Microsoft Azure Blob Storage Request node
• Amazon S3 Request node
• Microsoft Entra ID Input node
• Microsoft Entra ID Request node

To use an HTTP Proxy in conjunction with these message flow nodes, a Toolkit user should start by creating an HTTP Proxy policy within a policy project. The HTTP Proxy policy provides a URL (with hostname and port) for addressing the HTTP Proxy, and a Credential Name which is used to reference the credential to be used when connecting to the proxy.

Once the policy has been created and saved, and having placed one of the four supported message flow nodes into your flow, you should then start configuring the node in the usual way by clicking the Launch Connector Discovery button. The discovery process will continue as normal (selections for a policy project and the vault in which credentials will be stored), but when it comes to connecting to the endpoint application, you will find a new optional property in the dialog named Proxy name, as shown below.

As shown in the example, this field is used to specify the name of the policy project and the HTTP Proxy Policy it contains. When clicking connect, this policy will be accessed to retrieve the relevant required connection information in order to use the proxy. The proxy will be used both at discovery time, and the configuration will also be persisted into the message flow and used at runtime when data is sent through the message flow. For this reason, remember to include the HTTPProxy policy alongside your message flow when deploying to an integration server.

Runtime Monitoring Events Specify File Location

For a long time, App Connect Enterprise (and its predecessor products of different names such as IBM Integration Bus) have enabled users to publish monitoring events as messages pass through particular nominated locations in a message flow, such as from particular named terminals of particular message flow nodes. These monitoring events have traditionally been published using IBM MQ or the MQTT protocol, but they can also be written to a local file if you prefer. Until this mod release (ACE 12.0.12.0) when writing monitoring events to a file, the product has hard-coded the file path, number of files, and maximum size of each file using the following values:

• File Path: <work-dir>/config/common/monitoringEvents
• NumberOfFiles: 4
• SizeOfFile: 25MB

Starting from ACE 12.0.12.0 new options have been provided in the server.conf.yaml file which allows a user to select their own values to enable more detailed control of how this feature operates. Shown below is a relevant example stanza from the server.conf.yaml file:

Events:
BusinessEvents: # Monitoring events
  File:
    enabled: true        # Set true or false, default false 
    outputFormat: 'json' # Set comma separated list of one or more of : json,xml. Defaults to 'json'
    #filePath: ''        # If this is set to '' then the default path is /config/common/monitoringEvents. 
    #numberOfFiles: 8    # The maximum number of files that monitoring event file writing can rotate through. 
    #sizeOfFile: 50      # The maximum size in MB of a single file that a monitoring file can use before rotating to the next file.

Path Tracking in the Webui

ACE mod release 12.0.12.0 introduces a new feature in the Web UI that can help when diagnosing unexpected flow behavior in production environments. The purpose of the feature (when switched on, as the capability is not enabled by default) is to summarise the set of alternate paths that have been traversed by messages passing through a message flow. When working with complex flows, this provides a method of quickly drilling down into areas of interest when troubleshooting. For an enabled message flow, the Path analysis tab carries a tabular view showing identifiers for each path that has been recently traversed, and the Message count column shows the number of messages that have followed the given path.

There are three ways in which you can switch on the Path Tracking feature:

• Modify the server.conf.yaml to set the "enabled" property of the PathTracker Resource Manager to "true", as shown below:

  PathTracker:
    enabled: true # Enable path tracking for all message flows, default is false.

• Use the mqsichangeproperties command to set the "enabled" property of the PathTracker Resource Manager to true. For example for an integration node (named "MyIntegrationNode") owned server named "default"

 mqsichangeproperties MyIntegrationNode -e default -o PathTracker -n enabled -v true

• Use the Admin REST API to patch the "enabled" property of the PathTracker Resource Manager to "true". For example for an integration node-owned server named "default":

curl -v http://localhost:4414/apiv2/servers/default/resource-managers/path-tracker -X PATCH --data "{ \"properties\": { \"enabled\": true} }" -H "Content-Type: application/json"

Once the feature is enabled, as a message is propagated through the flow it stores context information in memory which is updated each time a new message flow node terminal is encountered. For performance reasons, this is stored on a per-thread basis. When the message flow completes this context information is converted into a completed path representation. At this point, a per-flow cache is interrogated to see if this path has been seen before. If the path is new then the path will be added to the cache and the message number will be incremented, adding to the message list for the path in question.

In order to ensure that the cache does not consume an unbounded amount of memory, the size of the cache is limited to 20 entries per flow, and for similar reasons, the message list is limited to 100 message IDs per path. Should the size of the cache reach the maximum of 20 entries, and a new path be observed, then the entry that was accessed least recently is evicted from the cache in order to make room for the new entry. The message list stores the most recent 100 entries for each path, if a new message is added the oldest entry is evicted from the message list.

This has a few consequences for how the data should be interpreted. Firstly it is possible for a path to drop out of the cache and then be added back to the cache at a later time because that path hasn't been traversed for a long time. At this point its message list will have been lost and message count reset, so the message counts should be viewed as a relative measure of path usage rather than an absolute number of messages processed. 

More detail on this new feature including a full worked example is provided in a dedicated blog entry which was recently published by David Crighton, Lead Architect for the App Connect Enterprise Level 3 Service team.

Improved Java 11 Support

ACE 12.0.12.0 continues our mission to enable more functionality of the integration server to work when using Java 11. You can create a work directory for an independent integration server and then configure it to use Java 11 using the following commands.

Create a working directory:

mqsicreateworkdir C:\temp\MyWorkDir
mqsicreateworkdir: Copying sample server.config.yaml to work directory
        1 file(s) copied.
Successful command completion.

Configure to use Java 11:

ibmint specify jre --work-directory C:\temp\MyWorkDir --version 11

If you navigate to the work directory of the server, you will find a server.java.yaml file will have been created which redirects to another file named embedded.java.yaml in the product's installation directory which specifies which java functionalities should remain disabled for the server. For ACE 12.0.12.0 the contents of this file are as shown below:

Whilst the file above shows that there are still several features that are not yet supported in servers that run under Java 11, there has been significant progress since the last mod release, with servers running Java 11 now supporting the deployment of message flows containing instances of the CICSRequest node, IMSRequest node, Mapping node, and ODMRules node. We intend to continue increasing the features supportive of execution under Java 11 in future releases.

Blocking the Retrieval of Credentials Stored in a Vault

ACE 12.0.12.0 introduces a small enhancement to the App Connect Enterprise vault technology. An App Connect Enterprise vault can be used to symmetrically encrypt and store credentials, which can then be used to access secured resources and endpoints from a message flow. Prior to this enhancement, a product administrator with local command console access on the machine where ACE is running, who is also in possession of the vault key, could execute the mqsivault command with a decode option if they wanted to retrieve named records from the vault and have them displayed on the screen. Consider the following example.

Create an external directory vault:

mqsivault --ext-vault-dir C:\temp\MyExtDirVault1 --ext-vault-key SuperSecret123 --create

Add a credential to the external directory vault:

mqsicredentials --ext-vault-dir C:\temp\MyExtDirVault1 --ext-vault-key SuperSecret123 --create --credential-type ftp --credential-name MyFTPIdentity --username ben --password myftppassword

Retrieve the credential from the vault and display it again:

mqsivault --ext-vault-dir C:\temp\MyExtDirVault1 --ext-vault-key SuperSecret123 --decode credentials/ftp/MyFTPIdentity

and the credentials will be displayed like this:

Namespace: credentials
Record: ftp/MyFTPIdentity
{"name":"MyFTPIdentity","type":"ftp","properties":{"authType":"basic","password":"myftppassword","username":"ben"}}
BIP8071I: Successful command completion.

Some users of ACE might not want the system administrator to have such wide-ranging powers, so from this latest mod release 12.0.12.0, when creating a vault you can specify --vault-options no-export and this will prevent the retrieval, display, and export of the credentials in a vault. Note that this request can only be specified when the vault is being created, and you cannot change options after the vault has been created. Compare the following example with the one above:

Create an external directory vault but this time use  --vault-options no-export:

mqsivault --ext-vault-dir C:\temp\MyExtDirVault2 --ext-vault-key SuperSecret123 --create --vault-options no-export

Add a credential to the external directory vault just like before:

mqsicredentials --ext-vault-dir C:\temp\MyExtDirVault2 --ext-vault-key SuperSecret123 --create --credential-type ftp --credential-name MyFTPIdentity --username ben --password myftppassword

Attempts to retrieve the credential from the vault and display it again will not work:

Namespace: credentials
Record: ftp/MyFTPIdentity
BIP15178E: Entries from this vault cannot be displayed due to vault options.
The vault was created with an option to prevent decode of its entries.

Summary

IBM App Connect Enterprise is constantly evolving to support professionals in completing their integration needs. By diving into some of the latest product features and describing concepts, we aim in this article to help you learn more about how to use them to solve your business problems.