Java Applications Log Message Analytics Using Splunk

Splunk is the most-used server for collecting data from different sources, indexing that collected data, analyzing the index data, and preparing reports based on the indexed data. The source of data may be from any environment. The source may be in a structured format or an unstructured format. Splunk provides some intelligent algorithms to understand the indexed data that are used in security, IoT machine logs, and application logs and to provide data analytics.

Splunk Data Sources

Splunk provides a wide range of plugins to support various resources. For example, to understand Windows event logs, it provides one plugin. For understanding Ubuntu machines logs, it provides different plugins. Based on the analysis of logs, Splunk will generate the number of times that the operating system is loaded, the average CPU load in each session, and how many different users logged into the system. It will generate the reports and it can also generate dashboard for live data streaming.

Scalability

Splunk will not depend upon any external services and it doesn't need any database support because Splunk internally uses a file system to store the indexed results. If the incoming data is increased, we can add another instance of Splunk server to index and store the data. Splunk will internally communicate and it will distribute the incoming data. While the searching, it will collect the data from all Splunk servers and display it to the user. If we enable redundancy, then incoming data is distributed to more than one Splunk server.

Some monitoring products only allow you to keep so many months', weeks', or even days’ worth of data. Others reduce the granularity of older events, compressing many data points into one because of capacity limits. The same is not true for Splunk. It can literally index hundreds of terabytes per day and keep practically unlimited amounts of data.

Customize and Extend Splunk With Preferred Technology

You can instantly convert dashboards into HTML5 and create custom user experiences in HTML5 and JavaScript with the Splunk Web Framework. Use Python, Java, JavaScript, Ruby, C#, and PHP SDKs to integrate Splunk capabilities into developer workflows. 

Developer Tools

Splunk provides plugins for Visual Studio for .NET-based applications; Eclipse and IntelliJ for Java-based applications; a splunk-logging plugin for JavaScript; and a plugin on top of NodeJS. We can also use the splunk-bunyan-logger plugin for JavaScript to collect HTTP events.

The Splunk SDK for Java ships with built-in support for IntelliJ for easy integration. SpringSource introduced the Spring Integration Extension for Splunk, making it easier for java developers using Spring to log directly to Splunk from their applications.

The Splunk Event Collector is the fastest way to send data to the Splunk Enterprises server and Splunk Cloud. HTTP Event Collector (HEC) enables users to send data over HTTP and HTTPS protocol directly to the Splunk Enterprise and Splunk Cloud from application code. HEC provides the one REST endpoint for accepting incoming data. In our application, we can write to send data to the REST endpoint. HEC is authorized using tokens, so we don’t need to hardcode the username/password details.

HTTP Event Collector provides a new way for developers to send application logging and metrics directly to Splunk Enterprise and Splunk Cloud via HTTP in a highly efficient and secure manner.

Enabling HTTP Event Collector

1. We can turn on HTTP Event Collector by enabling its endpoint. By default, HEC is not enabled.
2. Generate HEC Token.
3. On the client end, create a POST request to include authentication header or a key/value pair to include the HEC token.
4. POST data to the HEC token receiver.

For Splunk cloud instances, it is not enabled. We can raise a request to Splunk cloud to enable the HEC token.

Connecting to Splunk Server Using Splunk SDK

We can add the Splunk JAR to the project classpath or we can download splunk-sdk and extract the Splunk JAR from the splunk-sdk-java/dist folder.

The sample Java program below can be used to connect to Splunk Server and print the application on it.

importcom.splunk.*;
public class SplunkTest {
 public static void main(String[] args) {
  // Create a map of arguments and add login parameters
  ServiceArgsloginArgs = new ServiceArgs();
  loginArgs.setUsername("admin");
  loginArgs.setPassword("changeme");
  loginArgs.setHost("localhost");
  loginArgs.setPort(8089);
  // Create a Service instance and log in with the argument map
  Service service = Service.connect(loginArgs);
  // Print installed apps to the console to verify login
  for (Application app: service.getApplications().values()) {
   System.out.println(app.getName());
  }
 }
}

We can either user ServiceArgs or a Service constructor to pass host and port details.

importcom.splunk.*;
public class SplunkTest {
 public static void main(String[] args) {
  // A second way to create a new Service object and log in
  Service service = new Service("localhost", 8089);
  service.login("admin", "changeme");
  // Print installed apps to the console to verify login
  for (Application app: service.getApplications().values()) {
   System.out.println(app.getName());
  }
 }
}

Using the Same Token Across All Applications

The Service class creates a token after login. It will use this token for all subsequent requests.

importcom.splunk.*;
import com.sun.org.apache.xerces.internal.impl.dv.util.Base64;
public class SplunkTest {
 public static void main(String[] args) {
  Service service = new Service("localhost", 8089);
  String credentials = "admin:changeme";
  String basicAuthHeader = Base64.encode(credentials.getBytes());
  service.setToken("Basic " + basicAuthHeader);
  // Print the session token
  System.out.println("Your session token: " + service.getToken());
  // Print installed apps to the console to verify login
  for (Application app: service.getApplications().values()) {
   System.out.println(app.getName());
  }
 }
}

For example, if you want to use the same session token on multiple clients that are accessing a single Splunk Enterprise instance, you can log in on one client, get the session token using the Service.getToken method, and then pass that token to other clients to use when they log in (with the Service.setToken method). We can also connect to Splunk Enterprise by using basic access authentication — for example, if you have multiple Splunk Enterprise instances behind a load balancer and want to use the same session token for all. To create a token using a basic authorization header, Base64-encode a string with your credentials in the format "username:password" and prepend the authorization method ("Basic").

Integrating Splunk With Spring Applications

The Spring Integration adapter for Splunk includes two adapters:

1. Inbound Channel Adapter to search data from Splunk.
2. Outbound Channel Adapter to push event data into Splunk.

Spring integration provides a namespace for Splunk. It will use the Spring integration adapter for Splunk internally.

Creating beans for Splunk server object:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int-splunk="http://www.springframework.org/schema/integration/splunk"
xsi:schemaLocation="http://www.springframework.org/schema/integration/splunk
http://www.springframework.org/schema/integration/splunk/spring-integration-splunk.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<int-splunk:server id="splunkServer" host="somehost" port="8089" userName="user" password="password" owner="admin"/>
</beans>

Outbound Channel Adapter

Outbound channel adapter is used to put data into Splunk from channels in Spring Integration. There are three kinds of method to put data:

1. Submit (HTTP REST).

2. Stream.

3. TCP.

The main difference between using the REST inputs versus plain TCP/UDP inputs is really in the Splunk event handling pipeline.

With REST, you have to declare your event metadata (index, source, source type…) in the HTTP request at the source. You can't really transform the log event anymore after you have created it and sent it to Splunk. Typically, though, for people using REST, this is fine because they are well formatting their log events before sending them, anyway; no further processing/transforming and manipulation is required.

<int-splunk:outbound-channel-adapter id="splunkOutboundChannelAdapter" auto-startup="true" order="1" channel="outputToSplunk" splunk-server-ref="splunkServer" pool-server-connection="true" sourceType="spring-integration" source="example" ingest="submit">
</int-splunk:outbound-channel-adapter>

With TCP inputs, metadata manipulates and transforms the event data in Splunk before it gets indexed (using entries in props.conf/transforms.conf). The metadata (index, source, source type…) gets declared on the Splunk side when you establish the TCP/UDP input and can also be dynamically created, so essentially, you have a lot more control over the indexing of the event data.

<int-splunk:outbound-channel-adapter id="splunkOutboundChannelAdapter" auto-startup="true" order="1" channel="outputToSplunk" splunk-server-ref="splunkServer" ingest="tcp" tcpPort="9999"/>

To use outbound channel adapter with Stream, you can define the adapter as the following:

<int-splunk:outbound-channel-adapter id="splunkOutboundChannelAdapter" auto-startup="true" order="1" channel="outputToSplunk" splunk-server-ref="splunkServer" ingest="stream"/>

Inbound Channel Adapter

Inbound Channel Adapter is used to get data out of Splunk and put it into Spring Integration's channel. There are five ways to get data out of Splunk:

1. Blocking.
2. Nonblocking.
3. Saved search.
4. Realtime.
5. Export.

<int-splunk:inbound-channel-adapter id="splunkInboundChannelAdapter" auto-startup="true" search="search spring:example" splunk-server-ref="splunkServer" channel="inputFromSplunk" mode="blocking" initEarliestTime="-1d">
<int:poller fixed-rate="5" time-unit="SECONDS"/>
</int-splunk:inbound-channel-adapter>

...where mode is one of the above 5 types.

Using Log Frameworks to Log Into Splunk Servers

Splunk logging provides the JAR, which provides appenders for the most-used logger frameworks.

• java.util.logging handler for logging to Splunk REST endpoints, logging to Splunk Raw TCP Server Socket, and logging to Splunk HEC Endpoint.

• Log4j appender for logging to Splunk REST endpoints, logging to Splunk Raw TCP Server Socket, and logging to Splunk HEC Endpoint.

• Logbackappender for logging to Splunk REST endpoints, logging to Splunk Raw TCP Server Socket, and logging to Splunk HEC Endpoint.

• Log4j 2 appender for logging to Splunk HEC Endpoint and for config examples for TCP and UDP logging.

For example, if we want to use REST services, we can include the below configuration in logger.properties.

log4j.appender..splunk.logging.log4j.appender.SplunkRestAppender

log4j.appender.splunkrest.

log4j.appender.splunkrest.

log4j.appender.splunkrest.

log4j.appender.splunkrest.

log4j.appender.splunkrest.

log4j.appender.splunkrest.

log4j.appender.splunkrest.

log4j.appender.splunkrest.

log4j.appender.splunkrest.

log4j.appender.splunkrest.

This way, it will be logged automatically.

Conclusion

In this article, we have explained the uses of Splunk in Java. Splunk is the software to accept log messages, do indexes using log messages, and provide metrics using log messages. It provides the SDK and logging frameworks for connecting to Splunk Enterprises and Splunk Cloud. Spring Integration provides an adapter for contacting Splunk Sever and provides a namespace for creating Splunk-related beans.