Run Process Debug Tools in Containers, But Install Nothing

Ever need to debug your process in containers? Use strace, lsof, pstree, or anything you name it. But after login, you get a surprise: The tools are not installed!

So what will you do? Typically we have 3 different methods. Check it out, and discuss with me.

(Hint: You don’t have to install the tools at all.)

Start an Nginx container for an explanation. We will run tools against the Nginx process afterwards.

# pull docker image
docker pull nginx:alpine

# start container
docker run -t -d --privileged \
    -h nginxtest --name nginx-test \
    -p 8080:80 nginx:alpine

Check the Nginx process:

# verify nginx httpd service
curl http://localhost:8080

# check process pid
docker exec nginx-test ps -ef | grep nginx

Method 1: Debug From Inside the Container

Let’s say you want to “strace -p $nginx_pid.” But strace is not available in the nginx:alpine image.

Login and install. Yes, it will work. But just old school.

docker exec -u root -it nginx-test sh

# Install strace
apk --update add strace

# strace nginx process
strace -p 1
##   strace: Process 1 attached
##   rt_sigsuspend([], 8

Why is it old school? It will populate the env. Especially when the containers are in production mode.

The more packages you have installed, the more issues you will get.

Method 2: Debug From Docker Host

Linux containers share the same Linux kernel.

We can find the process id from docker host, then debug the process.

# get nginx process pid
root@denny:/tmp# ps -ef | grep nginx
root     27871 27834  0 22:00 pts/2    00:00:00 nginx: master process nginx -g daemon off;
systemd+ 27931 27871  0 22:00 pts/2    00:00:00 nginx: worker process
root     31324 27756  0 23:26 pts/1    00:00:00 grep --color=auto nginx

# Install tools in docker host
apt-get install -y strace

# Run tools from docker host
root@denny:/tmp# strace -p 27871
strace: Process 27871 attached
rt_sigsuspend([], 8

Method 3: Debug From an Ephemeral Container

Build a temporary image with tools installed.

# Dockerfile
cat > Dockerfile <<EOF
FROM alpine
RUN apk update && apk add strace
CMD ["strace", "-p", "1"]
EOF

# Build image
docker build -t strace .

Start a temporary container(nginx-test). Then debug the Nginx process by strace.

export test_conainter="nginx-test"
docker run -t --name strace-test \
  --pid=container:$test_conainter \
  --net=container:$test_conainter \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  strace

Don’t forget to destroy the container when you have finished your debugging.

Apparently, I like Method 3 the best. In this article, we use the strace tool. Surely we can support more tools like this.

Not so difficult as you thought. Right, my friend?

So why don’t you give it a try now? Or share this post with your colleagues?