Wrangling the Different Docker APIs

Kathy Thomas

Jun. 30, 15 · Interview

Likes (0)

Comment

Save

1.6K Views

[This article was written by Alex Harford.]

Docker APIs are a convenient way for your systems to talk to Docker infrastructure. But sometimes there are challenges associated with them. I've outlined in this blog the steps you need to take and the items you need to look out for when working with Docker APIs.

Initial Docker Setup

Ensure you have the latest Docker client installed. It should be v1.6 or newer.

[alexh:~/work] docker pull ubuntu
latest: Pulling from ubuntu

428b411c28f0: Pull complete
435050075b3f: Pull complete
9fd3c8c9af32: Pull complete
6d4946999d4f: Already exists
ubuntu:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.

Digest:  sha256:45e42b43f2ff4850dcf52960ee89c21cda79ec657302d36faaaa07d880215dd9
Status: Downloaded newer image for ubuntu:latest
[alexh:~/work] docker run -ti ubuntu /bin/bash
root@1092e8ca2ead:/# ps
  PID TTY              TIME CMD
        1 ?            00:00:00 bash
   14 ?            00:00:00 ps
root@1092e8ca2ead:/# exit
exit

Daemons, Registries, Hubs

The Docker registry is used to host docker images for download. In the most simple case, it can be a process serving static images. This would be a read-only registry supporting GET operations only. If you need something more complex, you need to use a Docker registry web service. You can [a target="_blank" href="http://www.activestate.com/blog/2014/01/deploying-your-own-private-docker-registry"]run your own private Docker registry or use the public official Docker Hub. The Docker Hub contains a Docker registry, but also includes other features, like user authentication. In our examples, we will run an unauthenticated Docker registry.

Setup

If you are using standard Docker images, most people will pull from the Docker Hub, which is a publically accessible Docker registry. However, a more complicated service may be talking to private Docker registries running different versions of the API. Let’s assemble a test environment with both versions of the docker registry API so we can see the different ways you can access it.

First, pull down two versions of the docker registry from the Docker Hub:

 docker pull registry:0.9.1
0.9.1: Pulling from registry
e9e06b06e14c: Pull complete
a82efea989f9: Pull complete
37bea4ee0c81: Pull complete
07f8e8c5e660: Pull complete
1f4ab7282e19: Pull complete
3c27027cdae8: Pull complete
7e0e5314436e: Pull complete
2696504d3685: Pull complete
012772dbb1c6: Pull complete
e24d9fce1d00: Pull complete
fd2726a79da8: Pull complete
bffc32d7113a: Pull complete
0cd49aa0e23c: Pull complete
4e698fa80441: Already exists
registry:0.9.1: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:98937757728eecbd72c9276bf711260aa29896f15217ce05be0562287e73232d
Status: Downloaded newer image for registry:0.9.1
[alexh:~/work] docker pull registry:2.0.1
2.0.1: Pulling from registry
39bb80489af7: Pull complete
df2a0347c9d0: Pull complete
7a3871ba15f8: Pull complete
a2703ed272d7: Pull complete
68769176e114: Pull complete
ab2ab59d7d1b: Pull complete
882ecee9f360: Pull complete 
40de65f8e79f: Pull complete
0c4f9c7d798f: Pull complete
ca29675fe853: Pull complete
89d10e9463e5: Pull complete
1a5aa415e484: Pull complete
3ea7a9e93b04: Pull complete
769d811a57fd: Pull complete
ae8a4a3af1aa: Pull complete
85cc9a791bb5: Pull complete
9cd2c8646022: Pull complete
048c32c549b9: Pull complete
cbbbda28c189: Pull complete
2602c005e534: Pull complete
136beb445cfa: Pull complete
0c5e5ef1d7da: Already exists
registry:2.0.1: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:0cd177d687589aff586aa2c66c64d1c25657b8d09cff9e1492192f496e7786c3
Status: Downloaded newer image for registry:2.0.1

The next step is to start them. We will start the v1 registry on port 5000, and the v2 registry on port 6000. The v1 registry occasionally fails when starting due to a lock file race condition, so tell it to restart if necessary.

[alexh:~/work] docker run -p 5000:5000 -d --restart=on-failure:3 registry:0.9.1
896c651b9bfa9780b14e3710d20428baab8497c30b9bc89946b192e1d1c145aa
[alexh:~/work] docker run -p 6000:5000 -d registry:2.0.1 e09d4204921c732879ee9b7544cd40a25275e0d1f1702cacd954412cfd586ffb
[alexh:~/work] docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                    NAMES
e09d4204921c        registry:2.0.1      "registry cmd/regist   4 seconds ago       Up 3 seconds        0.0.0.0:6000->5000/tcp   silly_albattani    
896c651b9bfa        registry:0.9.1      "docker-registry"      35 seconds ago      Up 34 seconds       0.0.0.0:5000->5000/tcp   jovial_leakey

Understanding Docker Namespaces

Docker has a concept of namespaces for its repositories which can be confusing.

[a target="_blank" href="https://docs.docker.com/docker-hub/official_repos/"]Official Repositories can be referred to without a username prefix:

Internally these are prefixed by library/. This means that command like docker pull ubuntu:15.10 and docker pull library/ubuntu:15.10 are equivalent.

If the name includes a '/' character (samalba/docker-registry), the left side refers to the username, and the right side refers to the image name in their public repository. It gets more complex when accessing private registries. The format becomes HOST:PORT/[USERNAME/]IMAGE. However, you should note that there is no authentication performed at this layer of our docker registry environment: anyone can push, pull, or delete from any 'user'. If the USERNAME is omitted, it is internally treated as being an 'official' image, and prefixed with library/.

 docker pull 127.0.0.1:5000/library/test-ubuntu
Pulling repository 127.0.0.1:5000/library/test-ubuntu
FATA[0004] Error: image library/test-ubuntu:latest not found
[alexh:~/work] docker tag 0fe5a10d2cf8 127.0.0.1:5000/test-ubuntu
[alexh:~/work] docker push 127.0.0.1:5000/test-ubuntu
The push refers to a repository [127.0.0.1:5000/test-ubuntu] (len: 1)
Sending image list
Pushing repository 127.0.0.1:5000/test-ubuntu (1 tags)
Image 5c1d0c04c3b8 already pushed, skipping
Image 8c63e4ac9a5f already pushed, skipping
Image 5fc05c0feaea already pushed, skipping
Image 0fe5a10d2cf8 already pushed, skipping
Pushing tag for rev [0fe5a10d2cf8] on {http://127.0.0.1:5000/v1/repositories/test-ubuntu/tags/latest}
[alexh:~/work] docker pull 127.0.0.1:5000/library/test-ubuntu
Pulling repository 127.0.0.1:5000/library/test-ubuntu
0fe5a10d2cf8: Download complete 
5c1d0c04c3b8: Download complete 
8c63e4ac9a5f: Download complete 
5fc05c0feaea: Download complete 
Status: Image is up to date for 127.0.0.1:5000/library/test-ubuntu:latest

In the v2 Docker registry, the [a target="_blank" href="https://docs.docker.com/registry/spec/api/#overview"]URI scheme has changed to allow the repository name to be broken up into multiple components. However, the Docker client does not yet support this flexibility. In the future, you should be able to extend the namespace of your registries, ie `redhat/centos/beta or redhat/fedora/stable.

Populating the Registries

We'll use Ubuntu 15.10 as our example image:

 docker pull ubuntu:15.10
15.10: Pulling from ubuntu
5c1d0c04c3b8: Pull complete
8c63e4ac9a5f: Pull complete
5fc05c0feaea: Pull complete
0fe5a10d2cf8: Already exists
ubuntu:15.10: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:d569b6ebfc62f35f9792392724bd4a74a4f5f5af10ccbc1880974ae2f0660898
Status: Downloaded newer image for ubuntu:15.10

It needs to be tagged with the new URL in order to push it to the private registries:

[alexh:~/work] docker tag ubuntu:15.10 127.0.0.1:5000/ubuntu:15.10
[alexh:~/work] docker tag ubuntu:15.10 127.0.0.1:6000/ubuntu:15.10
[alexh:~/work] docker push 127.0.0.1:5000/ubuntu:15.10
The push refers to a repository [127.0.0.1:5000/ubuntu] (len: 1)
Sending image list
Pushing repository 127.0.0.1:5000/ubuntu (1 tags)
5c1d0c04c3b8: Image successfully pushed
8c63e4ac9a5f: Image successfully pushed
5fc05c0feaea: Image successfully pushed
0fe5a10d2cf8: Image successfully pushed
Pushing tag for rev [0fe5a10d2cf8] on {http://127.0.0.1:5000/v1/repositories/ubuntu/tags/15.10}
[alexh:~/work] docker push 127.0.0.1:6000/ubuntu:15.10
The push refers to a repository [127.0.0.1:6000/ubuntu] (len: 1)
0fe5a10d2cf8: Image already exists
5fc05c0feaea: Image successfully pushed
8c63e4ac9a5f: Image successfully pushed
5c1d0c04c3b8: Image successfully pushed
Digest: sha256:1f93077ce8f2fa1da8aae87735f395eae93a1c21928d3e2d130717c9aeff177d

Note that the output between the v1 registry (on port 5000) and v2 (port 6000) are slightly different, but the result is the same: the Ubuntu image is now available on each registry.

Docker Registry APIs

At this point, we're able to compare the different APIs. In April 2015, Docker [a target="_blank" href="http://docs.docker.com/v1.6/release-notes/"]released version 1.6 and this included v2 of the Registry. Your software should be aware of the different versions of the Docker Registry API to handle these differences. Let's look at what it takes to download the image layers through the various APIs in order to make an offline cache.

First, we'll prepare our environment:

[alexh:~/work] export image=ubuntu
[alexh:~/work] export tag=15.10

v1

The v1 private registry can be examined at this point:

[alexh:~/work] curl -s http://127.0.0.1:5000/v1/repositories/library/$image/tags/$tag | python -m json.tool
"0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547"
export v1_image_id=`curl -s http://127.0.0.1:5000/v1/repositories/library/$image/tags/$tag | sed 's/"//g'`
[alexh:~/work] curl -s http://127.0.0.1:5000/v1/images/$v1_image_id/ancestry | python -m json.tool
[
   "0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547",
   "5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1",
   "8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824",
   "5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73"
]
[alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547/layer > 0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547.tar.gz
[alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1/layer > 5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1.tar.gz
[alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824/layer > 8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824.tar.gz
[alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73/layer > 5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73.tar.gz

v1 on Docker Hub

The Docker Hub currently implements the v1 API, but requires an authentication token for certain operations. It also allows multiple endpoints to be returned by the server. We'll take the simple approach of always using the first endpoint:

[alexh:~/work] export endpoint=`curl -sSL -o /dev/null -D- "https://index.docker.io/v1/repositories/$image/images" | awk '/X-Docker-Endpoints/{print $2}' | tr -d '\r' | sed 's/,//'`
[alexh:~/work] echo $endpoint
registry-1.docker.io
[alexh:~/work] export token=`curl -sSL -o /dev/null -D- -H 'X-Docker-Token: true' "https://index.docker.io/v1/repositories/$image/images" | tr -d '\r' | awk '/X-Docker-Token/{print $2}'`

The token needs to be used for authentication for the rest of the commands, but otherwise they are the same as the v1 private registry:

[alexh:~/work] export v1_image_id=`curl -s -H "Authorization: Token $token" https://$endpoint/v1/repositories/library/$image/tags/$tag | sed 's/"//g'`
[alexh:~/work] curl -sSL -H "Authorization: Token $token" "https://registry-1.docker.io/v1/images/$v1_image_id/ancestry" | python -m json.tool
[
   "0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547",
   "5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1",
   "8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824",
   "5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73"
]
[alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547/layer > 0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547.tar.gz
[alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1/layer > 5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1.tar.gz
[alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824/layer > 8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824.tar.gz
[alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73/layer > 5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73.tar.gz

v2 API

The v2 API works with manifest files that include checksums. It's also slightly simpler. A manifest file for a tag contains all of the layer information, rather than requiring an image ID to be looked up for a tag, and then the ancestry for that image to be looked up.

[alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/manifests/$tag | python -c 'import sys, json, pprint; pprint.pprint(json.load(sys.stdin)["fsLayers"])'
[{u'blobSum': u'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},
{u'blobSum': u'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},
{u'blobSum': u'sha256:d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d'},
{u'blobSum': u'sha256:23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca'},
{u'blobSum': u'sha256:7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107'}]
[alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 > a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz
[alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d > d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d.tar.gz
[alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca > 23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca.tar.gz
[alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107 > 7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107.tar.gz

We can get the checksum for these files to verify that they are what is described in the manifest file:

[alexh:~/work] sha256sum *.tar.gz
a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4  a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz
d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d  d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d.tar.gz
23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca  23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca.tar.gz
7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107  7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107.tar.gz

The Remote (daemon) API

Another API that is available is the Docker daemon running locally. It can be accessed over a Unix socket, or over TCP if the daemon is configured to allow it.

[alexh:~/work] echo -e "GET /images/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock | tail -n +6 | python -m json.tool
[
   {
       "Created": 1433116930,
       "Id": "0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547",
       "Labels": {},
       "ParentId": "5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1",
       "RepoDigests": [],
       "RepoTags": [
           "127.0.0.1:6000/ubuntu:15.10",
           "ubuntu:15.10",
           "127.0.0.1:5000/ubuntu:15.10"
       ],
       "Size": 0,
       "VirtualSize": 132392276
   },
   {
       "Created": 1432704049,
       "Id": "0c5e5ef1d7dac23c7164ea48faafc79f0c921f6cf87d2d8ea7469832ea31e4ca",
       "Labels": {},
       "ParentId": "136beb445cfa7f48dbe4e36a80a83d4b7945682827fd8bfb1510ac17b6a200c0",
       "RepoDigests": [],
       "RepoTags": [
           "registry:2.0.1"
       ],
       "Size": 0,
       "VirtualSize": 548626543
   },
   {
       "Created": 1432703977,
       "Id": "4e698fa804417b34b334793bab8a143403be9384e0651067b0c3933fe8d90eb2",
       "Labels": {},
       "ParentId": "0cd49aa0e23cfe176cbea4bf622d552a6f16b21965cf52d633f8c9e27438f52c",
       "RepoDigests": [],
       "RepoTags": [
           "registry:0.9.1"
       ],
       "Size": 0,
       "VirtualSize": 413940033
   }
]

A tarball containing all of the layers for a tag can be generated:

[alexh:~/work] echo -e "GET /images/get?names=$image:$tag HTTP/1.0\r\n" | nc -U /var/run/docker.sock | tail -n +5 > $image-$tag.tar
[alexh:~/work] mkdir tmp
[alexh:~/work] tar -C tmp -xf ubuntu-15.10.tar
[alexh:~/work] ls -l tmp
total 20
drwxr-xr-x 2 alexh alexh 4096 Jun  2 15:33 0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547
drwxr-xr-x 2 alexh alexh 4096 Jun  2 15:33 5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73
drwxr-xr-x 2 alexh alexh 4096 Jun  2 15:33 5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1
drwxr-xr-x 2 alexh alexh 4096 Jun  2 15:33 8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824
-rw-r--r-- 1 alexh alexh   87 Jun  2 15:33 repositories

Conclusions

Docker is a great technology and there are a lot of improvements and new features coming out at a rapid pace. Fortunately it's well documented and discussions about bugs are in the open on GitHub. However, there are still some edge cases to be aware of when talking to the Docker APIs. With some good design choices, your applications can be made backwards and forwards compatible, and will be able to use a wide range of Docker client versions and remote APIs.

Docker (software)

Opinions expressed by DZone contributors are their own.

Related

Trending