Export JSON Logs to ELK Stack

Centralized logging can be very useful when attempting to identify problems with your servers or applications, as it allows you to search through all of your logs in a single place. It is also useful because it allows you to identify issues that span multiple servers by correlating their logs during a specific time frame.

ELK Stack is very good tool for indexing logs. However, changing mappings can be a big headache. That's where JSON can become handy.

These days, JSON is a common format for exchanging data. It is both readable and reasonably compact, and it provides a standardized format for structuring data. Almost every programming language can parse it. JSON’s Unicode encoding makes it universally accessible, and its large and established user base provides an active community of helpful examples, patterns, and support.

In this blog post, we will examine the process of exporting your application JSON log into ELK stack.

1. Deploy an ELK Stack

If you don’t have one, you can deploy one using the Cloud 66 EasyDeploy App Store.

2. Config Your App to Log in JSON

Note: Sample below is in Rails.

# Gemfile
gem "lograge" 

Then:

# config/application.rb
# set up logging configuration
class Application < Rails::Application  
   Logging.configure(config)
end

Then:

# config/logging
require 'lograge'  
class Logging  
    def self.configure(config)
        # LOGGING!
        log_level = Rails.env.production? ? :info : :debug
        log_level_number = ::Logger.const_get(log_level.to_s.upcase)

        # LOGRAGE CONFIGURATION
        config.lograge.enabled = true
        config.lograge.formatter = Lograge::Formatters::Json.new

        # APP LOGGING
        logger = ::Logger.new(File.join(Rails.root, 'log', "#{Rails.env}.log"))
        logger.formatter = proc do |severity, datetime, progname, msg|
            lograge_hash = nil
            lograge_hash = JSON.parse(msg) rescue nil if msg.is_a?(String)
            msg = 'ActiveRecord' if lograge_hash
            log_hash = {
                time: datetime.utc.strftime('%Y-%m-%dT%H:%M:%S.%3NZ'),
                # hostname: hostname,
                pid: $$,
                thread_id: Thread.current.object_id.to_s(36),
                severity: severity,
                progname: progname,
                msg: msg.strip.force_encoding('UTF-8')
            }
            log_hash = log_hash.merge(lograge_hash) if lograge_hash
            json_string = nil
            begin
                json_string = "#{log_hash.to_json}\n"
            rescue
                # handle non-utf8 encodings
                log_hash[:msg] = msg.strip.force_encoding('ISO-8859-1').encode('UTF-8')
                json_string = "#{log_hash.to_json}\n"
            end
            json_string
        end

    tag_log = ActiveSupport::TaggedLogging.new(logger)
        config.logger = tag_log
        config.log_level = log_level
    end
end

You can find the full sample here.

3. Deploy/Run Your Application

With the sample above, you should see logs like below:

{"time":"2017-05-24T12:03:27.926Z","pid":22929,"thread_id":"ouypqenuo","severity":"INFO","progname":null,"msg":"","header":{"remote_ip":"92.168.124.59"}}

4. Install Filebeat on the Application Server

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ sudo apt-get install apt-transport-https
$ echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
$ sudo apt-get update && sudo apt-get install filebeat
$ sudo update-rc.d filebeat defaults 95 10

More info here.

5. Config Filebeat on the Application Server

Edit /etc/filebeat/filebeat.yml. 

filebeat.prospectors:  
- input_type: log
  paths:
    - <JSON_LOGFILE_PATH>
  json.keys_under_root: true
  json.overwrite_keys: false
  json.add_error_key: true
filebeat.registry_file: /var/lib/filebeat/registry  
output.logstash:  
  hosts: ["<YOUR_ELK_STACK_ADDRESS>:5044"]
  index: <YOUR_INDEX_NAME>

6. Start Filebeat

$ sudo /etc/init.d/filebeat start 

7. Config ELK Index Pattern

Change the index pattern to <YOUR_INDEX_NAME>-*. ELK will update fields base on logs that have been received.

Note: You can create different input_type when all of the logs aren't in JSON.

filebeat.prospectors:  
- input_type: log
  paths:
    - <JSON_LOGFILE_PATH>
  json.keys_under_root: true
  json.overwrite_keys: false
  json.add_error_key: true
- input_type: log
  paths:
    - <LOGFILES_PATH>/*.log
  exclude_files:
    - <JSON_LOGFILE>
  multiline.pattern: ^\[
  multiline.negate: true
  multiline.match: after
filebeat.registry_file: /var/lib/filebeat/registry  
output.logstash:  
  hosts: ["<YOUR_ELK_STACK_ADDRESS>:5044"]
  index: <YOUR_INDEX_NAME>

The biggest benefit of logging in JSON is that it’s a structured data format. This makes it possible for you to analyze your logs like Big Data. It’s not just readable text, but a database that can be queried. This allows summaries and analytics to take place and help you monitor your application and troubleshoot issues faster.