LogPacker: A New Log Management Platform

Greetings to everyone! By this matter, we start a series of articles about our product. LogPacker – is a log management system. Application daemon is distinguished by simplicity, reliability and resource efficiency. Now you don’t have to spend a lot of time for service setting and support, and also to create a great number of “kludges”. LogPacker already contains “kludge” solutions...

The main advantages of our service are integration with lots of software programs, fast installation, clustering, two layer aggregation, scanning function, minimum resources consumption, data speed and extra disk space.

LogPacker is a client-server architecture though any client can perform as a server and vice versa. In complex systems with big data, service nodes can be arranged in tree-structure architecture, where a big number of nodes are the clients (data transfer) and the servers (data process).

Our service is a perfect platform for data process and collection. Let’s consider the main functions of our service:

Scanning

Nobody wants to lose data. As a rule, when setting up log collector service, a configuration file is generated manually. At best configuration file is added to Puppet or Chef, however, there is still a big chance for data loss of key system components.

At first-run LogPacker automatically scan the server for logs and search for all data sources on the server. Then you have to select a list of logs that needs to be collected by editing configuration file. Scanning function allows you to save important data sources. At the same time, it allows you to handle configuration change yourself without restarting service. Scanning takes a few seconds.

Aggregation

In terms of aggregation client and server are identical. They process messages and convert to JSON format. This is two-layer client & server aggregation:

• Using group function in real time.
• Using flexible event sort in your system. Service is working only with certain types of messages.

Similar opportunities for client and server are made for interaction with unhandled data flows from clients who don’t have aggregation function. As for now we have created platform for the following opportunities:

• Log collection, transfer and process directly for mobile apps.
• JS error monitoring.
• Log monitoring and management for components, that don’t have client aggregation.

Aggregation function saves disk space and network resources, spread the load among the client and the server.

Reliable delivery

Any systems have problems with internal and external networks. Connection fault between the client and the server brings client data loss with no chance of recovery. Let’s consider the main service functions, which help to solve such problems and provide reliable message delivery:

• Automatic service restart upon failure
• Node health check
• Advanced connection supervision
• Automatic client backup is activated upon connection failure and other network errors. In the time of reconnection messages are sent to an application server or to the data storage.

As a result, LogPacker guarantees safe and reliable data delivery to all nodes.

Clustering

In the case of big data transfer and its processing, you can’t guarantee fault tolerance of the system when the system consists of one application server and a number of clients. If the single server fails, there will be a system crash and server data will be lost.

LogPacker cluster increases the system reliability, allows you to parallel data processing, automatically monitors and distributes servers load.

Let’s consider the main features of LogPacker cluster:

• Easy to add a new node to the cluster. You only have to inform new node of the existing node in the cluster. For that, you have to add a node to cluster.nodes (server.ini) of the running daemon. Restart is not required.
• Fast node deletes from the cluster, by just stopping the server.
• Node health check
• Cluster provides parallel insertion in different types of data storage for data rate display and its safety
• Automatic control and load balancing. Service controls the main servers’ parameters (CPUUser, CPUSys, MemFree, LADiskFree, LA) and distributes the load across cluster nodes.

Cluster solution shows the ability to endure high stress and parallel processing of a large number of data flows, which arrive from clients on servers, mobile devices, and js scripts. High reliability and instant speed of cluster at high loads are distinctive features of our service.

Data Sources

After setting up the service and automatic file configuration, select from the list those journals that daemon needs to collect and analyze. There is also a possibility to create configuration file manually by referring to software log file. There is no need in additional plugins for identification and aggregation of logs 3rd party applications. The service works out of the box. The service automatically identify, collect and aggregate log files for a great amount of software.

Please, see the list below:

A full list of applications you can find at our website in the resources section. In the near future, a possibility for collecting and aggregating JavaScript errors and mobile crashes will be added. It will allow you to work with main mobile platforms and browsers.

Data Outputs

First of all, you need to define architecture for log storage. According to your data amount and special requirements, identify in your architecture – main server storage, search service, and cache. Set up the service for concurrent write to different storage types. For receiving data use REST API with search capability and message filtering. LogPacker easily integrates with many 3rd party applications.

Let’s review some of them:

We have provided the list of the main and most important software our service works with. Integration with 3d party applications is very convenient and takes a few minutes.

Notifications

Service uses three types of notes for system event notification: slack, email, and SMS. For each type, there is a possibility to set up periods for informative messages individually (seconds, minutes, hours, days, weeks, months). Configuration takes a few minutes on client or server. Let’s take a look at configuration file notify.ini:

; Choose a way to notify about new logs
; Choose an interval for this and etc.
; providers can be comma-separated. Available: sendmail, slack, smtp
providers=sendmail
; interval in seconds
interval=3600
; log levels to include
levels=Fatal,Error
; tags to include. all by default
tags=*

[Sendmail]
; emails are comma separated
emails=

[Slack]
; slack token
token=
; channels are comma separated
channels=

[SMTP]
; emails are comma separated
emails=
; SMTP host
host=smtp.example.com
; SMTP login
login=foo@bar.com
; SMTP pass
pass=
; SMTP port
port=587
; Reply-To
replyto=robot@example.com

Exclusive possibility to set up personal notification for certain groups and users can decrease the amount of unimportant information and increase problem-solving speed. As an example, for monitoring group, “fatal” and “error” events are important for all possible systems, but for the development team all types of events are important although not from all systems.

Today we start our service with free licenses for a certain amount of servers. A couple of articles on problems we face in terms of implementing will be published soon. Service will also include certain open-source components.

Thank you for getting acquainted with our service!