Docker Centralized Logging With ELK Stack
Join the DZone community and get the full member experience.
Join For FreeAs your infrastructure grows, it becomes crucial to have a reliable centralized logging system. Log centralization is becoming a key aspect of a variety of IT tasks and provides you with an overview of your entire system.
The best solution is to aggregate the logs from all containers, which is enriched with metadata so that it provides you with better traceability options and comes with awesome community support. This is where the ELK Stack comes into the picture. ELK, also known as the Elastic stack, is a combination of modern open-source tools like ElasticSearch, Logstash, and Kibana. It is a complete end-to-end log analysis solution you can use for your system.
Each component has its defined role to play: ElasticSearch is best in storing the raw logs, Logstash helps to collect and transform the logs into a consistent format, and Kibana adds a great visualization layer and helps you to manage your system in a user-friendly manner.
In this guide, you will learn how to deploy ELK and start aggregating container logs. Here we are going to combine ELK with Filebeat to aggregate the container logs. For this, we are going to build a custom Docker image.
Step 1 - Configuring Filebeat
Let’s begin with the Filebeat configuration. First, you have to create a Dockerfile to create an image:
xxxxxxxxxx
$ mkdir filebeat_docker && cd $_
$ touch Dockerfile && nano Dockerfile
Now, open the Dockerfile
in your preferred text editor, and copy/paste below mentioned lines:
xxxxxxxxxx
FROM docker.elastic.co/beats/filebeat:7.5.1
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN mkdir /usr/share/filebeat/dockerlogs
RUN chown -R root /usr/share/filebeat/
RUN chmod -R go-w /usr/share/filebeat/
In filebeat_docker
directory, create a filebeat.yml
file that contains configuration for Filebeat. For this guide, we are going to use a minimal filebeat.yml
file.
xxxxxxxxxx
filebeat.inputs
type docker
containers
path"/usr/share/dockerlogs/data"
stream"stdout"
ids
"*"
cri.parse_flagstrue
combine_partialtrue
exclude_files'\.gz$'
processors
add_docker_metadata
host"unix:///var/run/docker.sock"
filebeat.config.modules
path $ path.config /modules.d/*.yml
reload.enabled false
output.logstash
hosts"127.0.0.1:5044"
log files
logging.level error
logging.to_files false
logging.to_syslog false
loggins.metrice.enabled false
logging.files
path /var/log/filebeat
name filebeat
keepfiles7
permissions0644
ssl.verification_mode none
Now, it’s time to create the Filebeat Docker image:
xxxxxxxxxx
$ docker build -t filebeatimage .
Sending build context to Docker daemon 3.584kB
Step 1/6 : FROM docker.elastic.co/beats/filebeat:7.5.1
7.5.1: Pulling from beats/filebeat
c808caf183b6: Already exists
a07383b84bc8: Pull complete
a3c8dd4531b4: Pull complete
5547f4a87d0c: Pull complete
d68e041d92cd: Pull complete
7cfb3f76a272: Pull complete
748d7fe7bf07: Pull complete
Digest: sha256:68d87ae7e7bb99832187f8ed5931cd253d7a6fd816a4bf6a077519c8553074e4
Status: Downloaded newer image for docker.elastic.co/beats/filebeat:7.5.1
---> 00c5b17745d1
Step 2/6 : COPY filebeat.yml /usr/share/filebeat/filebeat.yml
---> f6b75829d8d6
Step 3/6 : USER root
---> Running in 262c41d7ce58
Removing intermediate container 262c41d7ce58
---> 1ffcda8f39cf
Step 4/6 : RUN mkdir /usr/share/filebeat/dockerlogs
---> Running in 8612b1895ac7
Removing intermediate container 8612b1895ac7
---> 483d29e65dc7
Step 5/6 : RUN chown -R root /usr/share/filebeat/
---> Running in 4a6ad8b22705
Removing intermediate container 4a6ad8b22705
---> b779a9da7ac9
Step 6/6 : RUN chmod -R go-w /usr/share/filebeat/
---> Running in bb9638d12090
Removing intermediate container bb9638d12090
---> 85ec125594ee
Successfully built 85ec125594ee
Successfully tagged filebeatimage:latest
To verify if the image was built successfully:
xxxxxxxxxx
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
filebeatimage latest 85ec125594ee 7 seconds ago 514MB
For filebeat_elk
container, you have created two mounts using the parameter -v
;
/var/lib/docker/containers:/usr/share/dockerlogs/data
: You have mapped host machine docker logs which resides in/var/lib/docker/containers
to/usr/share/dockerlogs/data
inside the Docker container. Note that you have used:ro
which denotes that has read-only permission.- Whereas,
/var/run/docker.sock
is bound with Filebeat container’s Docker daemon, which allows Filebeat container to gather the Docker’s metadata and container logs entries.
Filebeat Installation via DEB
There is an alternate way to install Filebeat in your host machine. At the time of writing, Filebeat version is 7.5.1
you can download the latest version of Filebeat from here.
To install the downloaded .deb
file:
xxxxxxxxxx
$ wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.1-amd64.deb
$ sudo dpkg -i filebeat-7.5.1-amd64.deb
You can find the configuration file in /etc/filebeat/filebeat.yml
directory.
Step 2 - Configuring ELK
You can either use a remote server to host your ELK stack or can launch containers within your existing system.
Before you get going, make sure that the following ports are listening:
- Elasticsearch - Port
9200
and Port9300
- Logstash - Port
5044
- Kibana - Port
5601
ElasticSearch
We are going to use the latest official image of Elasticsearch as of now. So begin by pulling the image from Docker Hub:
xxxxxxxxxx
$ docker pull docker.elastic.co/elasticsearch/elasticsearch:7.5.1
7.5.1: Pulling from elasticsearch/elasticsearch
c808caf183b6: Already exists
05ff3f896999: Pull complete
82fb7fb0a94e: Pull complete
c4d0024708f4: Pull complete
136650a16cfe: Pull complete
968db096c092: Pull complete
42547e91692f: Pull complete
Digest: sha256:b0960105e830085acbb1f9c8001f58626506ce118f33816ea5d38c772bfc7e6c
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.5.1
docker.elastic.co/elasticsearch/elasticsearch:7.5.1
Now, create a directory name as docker_elk
, where all your configuration files and Dockerfile will reside:
$ mkdir docker_elk && cd $_
Inside docker_elk
, create another directory for elasticsearch
and create a Dockerfile
and elasticsearch.yml
files:
xxxxxxxxxx
$ mkdir elasticsearch && cd $_
$ touch Dockerfile && touch elasticsearch.yml
Open elasticsearch.yml
file in your preferred text editor and copy the configuration setting as it is:
xxxxxxxxxx
---
cluster.name"docker-cluster"
network.host0.0.0.0
xpack.license.self_generated.type basic
xpack.security.enabled true
xpack.monitoring.collection.enabledtrue
Note that you can set xpack.license.self_generated.type
from basic
to trial
if you wish to evaluate the commercial feature of x-pack for 30 days.
Open Dockerfile in your preferred text editor and copy the below-mentioned lines and paste it as it is:
xxxxxxxxxx
FROM docker.elastic.co/elasticsearch/elasticsearch:7.5.1
COPY --chown=elasticsearch:elasticsearch ./elasticsearch.yml /usr/share/elasticsearch/config/
The command chown
is to change the file owner to elasticsearch
as of other files in container.
Kibana
Now, you are going to setup Dockerfile for Kibana, and again you have to pull the latest image from the Elastic Docker registry:
xxxxxxxxxx
$ docker pull docker.elastic.co/kibana/kibana:7.5.1
7.5.1: Pulling from kibana/kibana
c808caf183b6: Already exists
e12a414b7b04: Pull complete
20714d0b39d8: Pull complete
393e0a5bccf2: Pull complete
b142626e938b: Pull complete
b28e35a143ca: Pull complete
728725922476: Pull complete
96692e1a8406: Pull complete
e4c3cbe1dbbe: Pull complete
bb6fc46a19d1: Pull complete
Digest: sha256:12b5e37e0f960108750e84f6b2f8acce409e01399992636b2a47d88bbc7c2611
Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.5.1
docker.elastic.co/kibana/kibana:7.5.1
Inside your docker_elk
, create a directory, and inside of it, you have to create a Dockerfile
and kibana.yml
files:
xxxxxxxxxx
$ mkdir kibana && cd $_
$ touch Dockerfile && touch kibana.yml
kibana.yml
will consist of follow configurations. Note that you have to change the values of elasticsearch.user
and elasticsearch.password
:
xxxxxxxxxx
---
server.name kibana
server.host"0"
elasticsearch.hosts "http://elasticsearch:9200"
xpack.monitoring.ui.container.elasticsearch.enabled true
elasticsearch.username elastic
elasticsearch.password yourstrongpasswordhere
Whereas, in Dockerfile
, will look something like this:
xxxxxxxxxx
FROM docker.elastic.co/kibana/kibana:7.5.1
COPY ./kibana.yml /usr/share/kibana/config/
Logstash
The container image for Logstash is available from the Elastic Docker registry. Again at the time of writing current version is 7.5.1, you can find latest version of Logstash here.
xxxxxxxxxx
$ docker pull docker.elastic.co/logstash/logstash:7.5.1
7.5.1: Pulling from logstash/logstash
c808caf183b6: Already exists
7c07521065ed: Pull complete
d0d212a3b734: Pull complete
418bd04a229b: Pull complete
b22f374f97b1: Pull complete
b65908943591: Pull complete
2ee12bfc6e9c: Pull complete
309701bd1d88: Pull complete
b3555469618d: Pull complete
2834c4c48906: Pull complete
bae432e5da20: Pull complete
Digest: sha256:5bc89224f65459072931bc782943a931f13b92a1a060261741897e724996ac1a
Status: Downloaded newer image for docker.elastic.co/logstash/logstash:7.5.1
docker.elastic.co/logstash/logstash:7.5.1
Now, create a directory for Logstash inside docker_elk
and add necessary files as shown below:
xxxxxxxxxx
$ mkdir logstash && cd $_
$ touch Dockerfile && touch logstash.yml
Copy below mentioned line into logstash.yml
. Make sure that you enter the right username and password in xpack.monitoring.elasticsearch.username
and xpack.monitoring.elasticsearch.password
respectively:
xxxxxxxxxx
---
http.host"0.0.0.0"
xpack.monitoring.elasticsearch.hosts "http://elasticsearch:9200"
xpack.monitoring.enabled true
xpack.monitoring.elasticsearch.username elastic
xpack.monitoring.elasticsearch.password yourstrongpasswordhere
Now, add following lines into your Dockerfile
:
xxxxxxxxxx
FROM docker.elastic.co/logstash/logstash:7.5.1
COPY ./logstash.yml /usr/share/logstash/config/
COPY ./logstash.conf /usr/share/logstash/pipeline/
Apart from this, you have to create a logstash.conf file
. Here in elasticsearch reference you will find host
, user
and password
, make sure you change the values as per your system:
xxxxxxxxxx
input {
tcp {
port => 5000
codec => json
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => elastic
password => yourstrongpasswordhere
}
}
As you are through with the setup of your stack's components, the directory structure of your project should should look something like this:
xxxxxxxxxx
.
├── elasticsearch
│ ├── Dockerfile
│ └── elasticsearch.yml
├── kibana
│ ├── Dockerfile
│ └── kibana.yml
└── logstash
├── Dockerfile
├── logstash.conf
└── logstash.yml
3 directories, 7 files
Now, it’s time to create a Docker Compose file, which will let you run the stack.
Step 3 - Docker Compose
Create a docker-compose.yml
file in the docker_elk
directory. Here you are going to define and run your multi-container application consist of Elasticsearch, Kibana, and Logstash.
You can copy the below-mentioned context in your docker-compose.yml file. Please make sure that you change the ELASTIC_PASSWORD
and ES_JAVA_OPTS
values. For this guide, ES_JAVA_OPTS
is set to 256 MB, but in real world scenarios you might want to increase the heap size as per requirement.
xxxxxxxxxx
version'3.2'
services
elasticsearch
build
context elasticsearch/
volumes
type volume
source elasticsearch
target /usr/share/elasticsearch/data
ports
"9200:9200"
"9300:9300"
environment
ES_JAVA_OPTS"-Xmx256m -Xms256m"
ELASTIC_PASSWORD yourstrongpasswordhere
discovery.type single-node
networks
elk_stack
logstash
build
context logstash/
ports
"5000:5000"
"9600:9600"
environment
LS_JAVA_OPTS"-Xmx256m -Xms256m"
networks
elk_stack
depends_on
elasticsearch
kibana
build
context kibana/
ports
"5601:5601"
networks
elk_stack
depends_on
elasticsearch
networks elk_stack
driver bridge
volumes
elasticsearch
Now, to build the ELK stack, you have to run the following command in your docker_elk
directory:
xxxxxxxxxx
$ docker-compose up -d
Starting elastic_elk ... done
Starting kibana_elk ... done
Starting logstash_elk ... done
To ensure that the pipeline is working all fine, run the following command to see the Elasticsearch indices:
xxxxxxxxxx
$ curl 'localhost:9200/_cat/indices?v' -u elastic:yourstrongpasswordhere
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .triggered_watches m-l01yMmT7y2PYU4mZ6-RA 1 0 0 0 6.5kb 6.5kb
green open .watcher-history-10-2020.01.10 SX3iYGedRKKCC6JLx_W8fA 1 0 1523 0 2mb 2mb
green open .management-beats ThHV2q9iSfiYo__s2rouIw 1 0 6 1 40.5kb 40.5kb
green open .ml-annotations-6 PwK7Zuw7RjytoWFuCCulJg 1 0 0 0 283b 283b
green open .monitoring-kibana-7-2020.01.10 8xVnx0ksTHShds7yDlHQvw 1 0 1006 0 385.4kb 385.4kb
green open .monitoring-es-7-2020.01.10 CZd89LiNS7q-RepP5ZWhEQ 1 0 36412 340 16.4mb 16.4mb
green open .apm-agent-configuration e7PRBda_QdGrWtV6KECsMA 1 0 0 0 283b 283b
green open .ml-anomalies-shared MddTZQ7-QBaHNTSmOtUqiQ 1 0 1 0 5.5kb 5.5kb
green open .kibana_1 akgBeG32QcS7AhjBOed3LA 1 0 1105 28 687.1kb 687.1kb
green open .ml-config CTLI-eNdTkyBmgLj3JVrEA 1 0 22 0 56.6kb 56.6kb
green open .ml-state gKx28CMGQiuZyx82bNUoYg 1 0 0 0 283b 283b
green open .security-7 krH4NlJeThyQRA-hwhPXEA 1 0 36 0 83.6kb 83.6kb
green open .logstash 7wxswFtbR3eepuWZHEIR9w 1 0 0 0 281b 281b
green open .kibana_task_manager_1 ft60q2R8R8-nviAyc0caoQ 1 0 2 1 16.2kb 16.2kb
yellow open filebeat-7.5.1-2020.01.10-000001 1-RGhyG9Tf-wGcepQ49mmg 1 1 0 0 283b 283b
green open .monitoring-alerts-7 TLxewhFyTKycI9IsjX0iVg 1 0 6 0 40.9kb 40.9kb
green open .monitoring-logstash-7-2020.01.10 dc_S5BhsRNuukwTxbrxvLw 1 0 4774 0 1.1mb 1.1mb
green open .watches x7QAcAQZTrab-pQuvonXpg 1 0 6 6 120.2kb 120.2kb
green open .ml-notifications-000001 vFYzmHorTVKZplMuW7VSmw 1 0 52 0 81.6kb 81.6kb
Now, it is time to pay a visit to our Kibana dashboard. Open your browser and enter the URL http://your-ip-addr-here:5601
. Now enter the predefined username and password; in our case, it is elastic
and yourstrongpasswordhere
, respectively.
In your Kibana dashboard, go to the Management tab, and under Kibana, click on Index Patterns. In the first row, you will find the filebeat-*
index, which already has been identified by Kibana.
Now, go to the Discover tag on the Kibana dashboard and view your container logs along with the metadata under the selected index pattern, which could look something like this:
Conclusion
You have now installed and configured the ELK Stack on your host machine, which is going to collect the raw log from your Docker into the stack that later can be analyzed or can be used to debug applications.
Published at DZone with permission of Sudip Sengupta. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments