Enable Custom Logging For MuleSoft Application Using Log4J2 Splunk HTTP and HTTP Appender

Logging is an important part of your environment for visualizing the data, troubleshooting, debugging and identifying the production issues quickly.

Jitendra Bafna

CORE ·

Updated Apr. 07, 20 · Tutorial

Likes (9)

Comment

Save

26.9K Views

Introduction

Logging is a very important part of your environment for visualizing the data, troubleshooting, debugging and identifying the production issues quickly.

Logging must be robust, consistent and reliable and most importantly that we can have some centralized and easy to use tools that can be used for logging purposes. There are various logging tools like Splunk, ELK, etc.

MuleSoft provides its logging framework for storing applications and system logs. The main reason for using a custom logging framework is that MuleSoft in Cloudhub stores data for 30 days or up to 100 MB.

Once the application is deleted, we will lose all the logs. So we required logging tools like Splunk for storing data and can persist for a longer time.

Splunk logging tool can be used to enable log MuleSoft On-Premise Runtime logs, On-Premise MuleSoft Application and CloudHub Application logs.

In the last article, we have seen how to set up Splunk and enable logging for MuleSoft Application. Click here for more details.

Enable Logging For CloudHub Application

Before enabling the logging for Cloudhub application, you need Disable CloudHub logs. By default, this option is not available and you need to raise a ticket with MuleSoft for providing this option.

cloudhub

Once you disabled CloudHub logs, MuleSoft is not responsible for below things

MuleSoft is not responsible for lost logging data due to the misconfiguration of your log4j appender.
MuleSoft is also not responsible for misconfigurations that result in performance degradation, running out of disk space, or other side effects.
When you disable the default CloudHub application logs, then only the system logs are available. For application worker logs, please check your own application’s logging system. Downloading logs is not an option in this scenario.
Only Asynchronous log appenders can be used, Synchronous appenders should not be used.
Use asynchronous loggers and not synchronous ones is to avoid threading issues. Synchronous loggers can lock threads waiting for responses.

You need to create log4j2.xml at location src/main/resources.

log4j2.xml

There are two ways you can send logs to Splunk

HTTP Appender
Splunk HTTP

We will see how we can send logs from CloudHub application to Splunk using Splunk HTTP.

You will be requiring the token and connection details related to Splunk. So you can go through the article and understand how to fetch those details.

You need to add a SplunkHttp appender in log4j.xml. Provide URL to connect Splunk, source, pattern layout, etc.

     XML 
   
xxxxxxxxxx
 
<SplunkHttp name="SPLUNK-CLOUD" source="mule-cloudhub-app"
            url="http://localhost:8088/" 
            token="b3a34eabd-b5f4-471431-97d2-e5b234af5bae9e" index="main"
            disableCertificateValidation="true">
            <PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
</SplunkHttp>

You need to make sure the logger must be asynchronous and it is recommended by MuleSoft also.

While using SplunkHttp, you need to add the following dependency in pom.xml of your application.

     XML 
   
xxxxxxxxxx
 
<dependency>
    <groupId>com.splunk.logging</groupId>
    <artifactId>splunk-library-javalogging</artifactId>
    <version>1.7.3</version>
</dependency>

You may require to add below two additional dependencies.

     XML 
   
xxxxxxxxxx
 
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.10.0</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.10.0</version>
</dependency>

You will be also required to add a repository into your pom.xml of your application.

     XML 
   
xxxxxxxxxx
 
<repository>
    <id>splunk-artifactory</id>
    <name>Splunk Releases</name>
    <url>http://splunk.jfrog.io/splunk/ext-releases-local</url>
</repository>

You can add more loggers like Log4J2CloudhubLogAppender into your log4j2.xml to enable logging on the CloudHub log console of your application.

     XML 
   
xxxxxxxxxx
 
<Log4J2CloudhubLogAppender name="CLOUDHUB"
            addressProvider="com.mulesoft.ch.logging.DefaultAggregatorAddressProvider"
            applicationContext="com.mulesoft.ch.logging.DefaultApplicationContext"
            appendRetryIntervalMs="${sys:logging.appendRetryInterval}"
            appendMaxAttempts="${sys:logging.appendMaxAttempts}"
            batchSendIntervalMs="${sys:logging.batchSendInterval}"
            batchMaxRecords="${sys:logging.batchMaxRecords}" memBufferMaxSize="${sys:logging.memBufferMaxSize}"
            journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"
            journalMaxFileSize="${sys:logging.journalMaxFileSize}"
            clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"
            clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"
            clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"
            serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"
            serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"
            statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">
            <PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
</Log4J2CloudhubLogAppender>

Below is full log4j2.xml which can be used for your application for enabling custom logging on CloudHub and Splunk.

     XML 
   
 
 
      
    
 
      
 
     
 
      
 
     
 
     
 
    
 
          x 
         
 
          
 
         
 
           
 
          
 
          
 
         
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO" name="cloudhub"
    packages="com.mulesoft.ch.logging.appender,com.splunk.logging,org.apache.logging.log4j">
    <Appenders>
        <Log4J2CloudhubLogAppender name="CLOUDHUB"
            addressProvider="com.mulesoft.ch.logging.DefaultAggregatorAddressProvider"
            applicationContext="com.mulesoft.ch.logging.DefaultApplicationContext"
            appendRetryIntervalMs="${sys:logging.appendRetryInterval}"
            appendMaxAttempts="${sys:logging.appendMaxAttempts}"
            batchSendIntervalMs="${sys:logging.batchSendInterval}"
            batchMaxRecords="${sys:logging.batchMaxRecords}" memBufferMaxSize="${sys:logging.memBufferMaxSize}"
            journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"
            journalMaxFileSize="${sys:logging.journalMaxFileSize}"
            clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"
            clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"
            clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"
            serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"
            serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"
            statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">
            <PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" ></PatternLayout>
        </Log4J2CloudhubLogAppender>
        <SplunkHttp name="SPLUNK-CLOUD" source="mule-cloudhub-app"
            url="http://localhost:8088/" 
            token="b3a34eabd-b5f4-471431-97d2-e5b234af5bae9e" index="main"
            disableCertificateValidation="true">
            <PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" ></PatternLayout>
        </SplunkHttp>
    </Appenders>
    <Loggers>
        <AsyncLogger
            name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor"
            level="INFO" ></AsyncLogger>
        <AsyncLogger name="com.mulesoft.agent" level="INFO" ></AsyncLogger>
        <AsyncRoot level="INFO">
            <AppenderRef ref="SPLUNK-CLOUD" ></AppenderRef>
            <AppenderRef ref="CLOUDHUB" ></AppenderRef>
        </AsyncRoot>
        <AsyncLogger name="com.gigaspaces" level="ERROR" ></AsyncLogger>
        <AsyncLogger name="com.j_spaces" level="ERROR" ></AsyncLogger>
        <AsyncLogger name="com.sun.jini" level="ERROR" ></AsyncLogger>
        <AsyncLogger name="net.jini" level="ERROR" ></AsyncLogger>
        <AsyncLogger name="org.apache" level="WARN" ></AsyncLogger>
        <AsyncLogger name="org.apache.cxf" level="WARN" ></AsyncLogger>
        <AsyncLogger name="org.springframework.beans.factory"
            level="WARN" ></AsyncLogger>
        <AsyncLogger name="org.mule" level="INFO" ></AsyncLogger>
        <AsyncLogger name="com.mulesoft" level="INFO" ></AsyncLogger>
        <AsyncLogger name="org.jetel" level="WARN" ></AsyncLogger>
        <AsyncLogger name="Tracking" level="WARN" ></AsyncLogger>
        <AsyncLogger name="org.mule" level="INFO" ></AsyncLogger>
        <AsyncLogger name="com.mulesoft" level="INFO" ></AsyncLogger>
        <AsyncLogger name="org.mule.extensions.jms" level="INFO" ></AsyncLogger>
        <AsyncLogger name="org.mule.service.http.impl.service.HttpMessageLogger"
            level="INFO" ></AsyncLogger>
        <AsyncLogger name="org.mule.extension.salesforce" level="INFO" ></AsyncLogger>
        <AsyncLogger name="org.mule.extension.ftp" level="INFO" ></AsyncLogger>
        <AsyncLogger name="org.mule.extension.sftp" level="INFO" ></AsyncLogger>
        <AsyncLogger name="com.mulesoft.extension.ftps" level="INFO" ></AsyncLogger>
        <AsyncLogger name="org.mule.modules.sap" level="INFO" ></AsyncLogger>
        <AsyncLogger name="com.mulesoft.extension.mq" level="INFO" ></AsyncLogger>
        <AsyncLogger name="com.mulesoft.mq" level="INFO" ></AsyncLogger>
        <AsyncLogger name="org.mule.extension.db" level="INFO" ></AsyncLogger>
        <AsyncLogger name="httpclient.wire" level="DEBUG" ></AsyncLogger>
        <AsyncLogger name="org.mule.transport.email" level="DEBUG" ></AsyncLogger>
    </Loggers>
</Configuration>
 
      
 
     

JSON Logger

JSON Logger is a component for logging the information, warning, errors. It logs the data in the form of JSON. It is recommended to use JSON Logger for your application and it logs the data in a well-formed way in any tools like Splunk, ELK, etc.

You can install JSON Logger in Anypoint Studio from Exchange.

Logger

Configure the JSON Logger as shown below.

JSON Logger

You can set the application name, application version and environment in the properties file.

     XML 
   
 
 
      
    
 
      
 
     
 
      
 
     
 
     
 
    
x
 
          
 
         
 
          
 
         

              
          
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
    xmlns:json-logger="http://www.mulesoft.org/schema/mule/json-logger"
    xmlns:api-gateway="http://www.mulesoft.org/schema/mule/api-gateway"
    xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd 
    http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
    http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd 
http://www.mulesoft.org/schema/mule/api-gateway http://www.mulesoft.org/schema/mule/api-gateway/current/mule-api-gateway.xsd
http://www.mulesoft.org/schema/mule/json-logger http://www.mulesoft.org/schema/mule/json-logger/current/mule-json-logger.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
    <http:request-config name="HTTP_Request_configuration" doc:name="HTTP Request configuration" doc:id="4dea45bf-d0c6-4ee8-baa0-0b8a9e508c88" basePath="/api/timezone" >
        <http:request-connection host="worldtimeapi.org" />
    </http:request-config>
    <configuration-properties doc:name="Configuration properties" doc:id="9dacbbe5-0ec7-4ce4-b094-ff89131a07cc" file="cloud.properties" />
    <global-property doc:name="Global Property" doc:id="7a60a2c8-efcf-4538-81c8-64248fe0dec4" name="mule.env" value="dev" />
    <http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="694e8fba-882a-4f04-805b-50c66ecec1a3" >
        <http:listener-connection host="0.0.0.0" port="${http.private.port}" />
    </http:listener-config>
    <json-logger:config name="JSON_Logger_Config" doc:name="JSON Logger Config" doc:id="59e12b0a-d64d-4300-af40-23b29f11c2dc"/>
    <flow name="test-application-cloudFlow" doc:id="7db040d5-7309-4688-a4c5-fea0f43c6103">
        <http:listener doc:name="Listener" doc:id="da11b76c-b4d5-4140-a539-5841c4690b8c" config-ref="HTTP_Listener_config" path="/v1.0/timezone/{region}/{city}">
            <http:error-response statusCode="#[vars.httpStatusCode]" >
                <http:body ><![CDATA[#[payload]]]></http:body>
            </http:error-response>
        </http:listener>
        <http:request method="GET" doc:name="Request" doc:id="730aec0d-fb82-4430-8cf3-c14264a7e9ae" config-ref="HTTP_Request_configuration" path="/{region}/{city}">
            <http:uri-params ><![CDATA[#[output application/java
---
{
    city : attributes.uriParams.city,
    region : attributes.uriParams.region
}]]]></http:uri-params>
        </http:request>
        <json-logger:logger doc:name="Logger" doc:id="f2257ace-a8b0-4f5f-83d5-4ac8a710983f" config-ref="JSON_Logger_Config" message="#['Information']"/>
        <error-handler >
            <on-error-propagate enableNotifications="true" logException="true" doc:name="On Error Propagate" doc:id="1d5f427f-ceaa-4af1-8e2d-2be0aaf45b46" type="HTTP:FORBIDDEN">
                <ee:transform doc:name="Transform Message" doc:id="b978dae0-7538-4910-82ee-3c12236b6603" >
                    <ee:message >
                        <ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
{
    error:error.description
}]]></ee:set-payload>
                    </ee:message>
                    <ee:variables >
                        <ee:set-variable variableName="httpStatusCode" ><![CDATA[403]]></ee:set-variable>
                    </ee:variables>
                </ee:transform>
                <json-logger:logger doc:name="Logger" doc:id="3c5e26da-4183-4318-9f6a-a07b86b717ff" config-ref="JSON_Logger_Config" message="Data From Timezone" priority="ERROR"/>
            </on-error-propagate>
        </error-handler>
    </flow>
</mule>
 
      
 
     

Now, you can verify the logs in Splunk and it will be in JSON format.

Enable Splunk Logging For On-Premise MuleSoft Runtime

For enabling the logging for On-Premise MuleSoft Runtime, you can use HTTP Appender.

You can add HTTP Appender in log4j2.xml located at $MULE_HOME/conf/

     XML 
   
          x 
         
<Http name="Splunk" url="http://localhost:8088/services/collector/raw">
            <Property name="Authorization" value="Splunk eert8e102-5f55-40ad-a047-b55a82245419"></Property>
            <PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n"></PatternLayout>
</Http>

This will allows sending logs to Splunk.

Full log4j2.xml

     XML 
   
          x 
         
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n" />
        </Console>
        <Console name="ConsoleLogUtil" target="SYSTEM_OUT">
            <PatternLayout pattern="%m%n" />
        </Console>
        <Http name="Splunk" url="http://localhost:8088/services/collector/raw">
            <Property name="Authorization" value="Splunk eert8e102-5f55-40ad-a047-b55a82245419"></Property>
            <PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n"></PatternLayout>
        </Http>
    </Appenders>
    <Loggers>
        <AsyncLogger
            name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor"
            level="INFO" />
        <AsyncLogger name="com.mulesoft.agent" level="INFO" />
        <AsyncLogger name="org.mule.runtime.core.internal.logging"
            additivity="false" level="INFO">
            <AppenderRef ref="ConsoleLogUtil" />
        </AsyncLogger>
        <AsyncRoot level="INFO">
            <AppenderRef ref="Console" />
            <AppenderRef ref="Splunk" />
        </AsyncRoot>
        <AsyncLogger name="com.mulesoft.mule.runtime.gw" level="INFO" />
        <AsyncLogger name="com.mulesoft.extension.policies" level="INFO" />
        <AsyncLogger name="com.mulesoft.extension.gateway" level="INFO" />
        <AsyncLogger name="com.mulesoft.analytics" level="INFO" />
        <AsyncLogger name="org.mule" level="INFO" />
        <AsyncLogger name="com.mulesoft" level="INFO" />
    </Loggers>
</Configuration>

This is how you can enable Splunk logging using SplunkHTTP and HTTP Appender for MuleSoft applications.

application MuleSoft

Opinions expressed by DZone contributors are their own.

Related

Trending

Enable Custom Logging For MuleSoft Application Using Log4J2 Splunk HTTP and HTTP Appender

Logging is an important part of your environment for visualizing the data, troubleshooting, debugging and identifying the production issues quickly.

Introduction

Enable Logging For CloudHub Application

JSON Logger

Enable Splunk Logging For On-Premise MuleSoft Runtime

Related

Partner Resources