Mount AWS EFS, NFS or CIFS/Samba volumes in Docker

Recently I had a project where I was dealing with large file systems that had to be spanned and preserved against hundreds of Docker services for computation.  I decided to use Amazon's EFS which was recently announced at re:Invent 2015.

I setup the file system which was easy and attempted to play with it.  I first mounted it manually via NFS4 on one of the Mesos slaves.  It worked and initial tests showed great performance.   Now
the next step was meeting some of my personal requirements before I could call this successful.

My Requirements

• The file system should be mounted when needed within a Docker container and tore down when no longer in use
• EFS provides endpoints in each availability zone so I also needed to support dynamic linking based on which slave the container was running in and the corresponding endpoint matching the AZ.
• Must use NFS4 (This is what EFS supports)
• Did not want to create or modify existing container images to attempt to mount NFS within the container itself.  This also would mean the container would need to run in privileged mode which I’m against

After analysis, I decided the best choice was to write a plugin myself to extend the Docker engine.

Docker Volume Plugins

Docker volume plugins enable Docker deployments to be integrated with external storage systems. A volume plugin makes use of the -v and --volume-driver flag on the docker run command. For example:

$ docker run -ti -v volumename:/data --volume-driver=some_plugin ubuntu sh

This command passes the volumename through to the volume plugin as a user-given name for the volume.

By having the user specify a volumename, a plugin can associate the volume to an external volume beyond the lifetime of a single container or container host. This can be used, for example, to move a stageful container from one server to another.

Meet Docker-Volume-Netshare Plugin

Before I began coding a plugin I wanted to pre-define some requirements. Here’s some of them below:

• Write it in Go so it runs at native speed
• Support NFS 3/4 and CIFS (might come in handy someday)
• For EFS support the plugin will lookup local EC2 metadata on the slave to find the corresponding AZ and region for building the URI from the File System ID
• Must support volume management
  • Understand when a volume is already mounted and keep track of its connections
  • If a container is stopped determining based on active connections if the volume mount should be removed
• Support all of the Docker Plugin API

After designing the plugin I installed it on the slaves (container hosts) and as a result I was able to dynamically mount EFS volumes and link them into the container based on the specified target mount.

Using Netshare:

First thing to do is download the binary for your particular platform at: https://github.com/gondor/docker-volume-netshare

The examples below are just a few and the plugin offers many options.  Let’s start with EFS.

Mounting EFS Volumes

The example below will start a container and mount the specified EFS File System by the FS_ID (assigned by AWS) into a directory called /data in the container.

1. Run the plugin or add it to systemd

   $ sudo docker-volume-netshare efs

2. Run a container to test:

   $ docker run -i -t --volume-driver=efs -v aws_fs_id:/data ubuntu /bin/bash

Mounting CIFS/Samba Volumes

The example below will start a container and mount the specified CIFS server/sharename into a directory called /data in the container.

1. Run the plugin or add it to systemd

   $ sudo docker-volume-netshare cifs --username user --password pass --domain domain

2. Run a container to test:

   $ docker run -i -t --volume-driver=cifs -v hostname/share:/data ubuntu /bin/bash

Mounting NFS Volumes

1. Run the plugin or add it to systemd

   $ sudo docker-volume-netshare nfs

2. Run a container to test:

   $ docker run -i -t --volume-driver=nfs -v hostname/volume:/data ubuntu /bin/bash

Whats Next…

Here are some future enhancements I will be adding to the plugin based on requests and priority:

• Support for credential files based on hosts or possibly a credential service
• REST service to show stats on mounted volumes and how many connections
• RPM packages for enterprise linux
• Systemd and init based scripts

For more information see full documentation at: https://github.com/gondor/docker-volume-netshare