Five Best Practices for GoLang CI/CD

±For developers programming in long-established languages like Java, JavaScript, or Python, the best way to build continuous integration and delivery (CI/CD) workflows with Artifactory is pretty familiar. A mature set of dependency management systems for those languages and container solutions, such as Docker, provide a clear roadmap.

But if you’re programming your applications in GoLang, how hard is it to practice CI/CD with the same kind of efficiency?

As it turns out, it’s gotten easier, especially with some of the latest innovations in Go. 

Best Practices

At JFrog, we’re big fans of GoLang, as we use it as the language for several of our flagship solutions. We practice what we promote, too, using Artifactory at the heart of our CI/CD. Here are some of the practices we can recommend:

1. Use Go Modules

Unlike many established programming languages, initial releases of GoLang didn’t provide a common mechanism for managing versioned dependencies. Instead, the Go team encouraged others to develop add-on tools for Go package versioning.

That changed with the release of Go 1.11 in August 2018, with support for Go modules. Now, the native dependency management solution for GoLang, Go modules are collections of related Go packages that are versioned together as a single unit. This enables developers to share code without repeatedly downloading it. 

A Go module is defined by a go.mod file in the project’s root directory, which specifies the module name along with its module dependencies. The module dependency is represented by the module name and version number.

If you haven’t adopted Go modules yet, you’ll need to follow the steps below:

1. Use go mod init to generate a go.mod file if previous package managers were used.
2. Use go mod tidy if other package managers were not used. This command will generate a populated go.mod file.
3. If the version is v2 and above, you will need to change the module name to add the corresponding suffix, update to the import path, add module aware static analysis tools, and finally update code generator files, such as .proto files to reflect the new import path.

For example, here is a go.mod file for a publicly available Go module for a structured logger:

module github.com/sirupsen/logrus
require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/konsorten/go-windows-terminal-sequences v1.0.1
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/stretchr/objx v0.1.1 // indirect
github.com/stretchr/testify v1.2.2
golang.org/x/sys v0.0.0-20190422165155-953cdadca894
)

Note that the version numbers must conform to semver convention (for example, v1.2.1 instead of 20190812, or 1.2.1) as required by the go command. You should avoid using pseudo versions like the one shown above (v0.0.0-yyyymmddhhmmss-abcdefabcdef) — although commit hash pseudo-version were introduced to bring Go Modules support to untagged projects, they should be only used as a fallback mechanism. If the dependencies you need have release tags, use those tags in your required statements.

In your own code, you can import this Go module along with other dependencies:

import (
"fmt"
"io/ioutil"
"net/http"
"os"
// Public Go Module for logging
log "github.com/sirupsen/logrus"
)

Then you can reference the module functions in your GoLang code:

 // Send text to the log
log.Printf("Hello Log!")

2. Use GOPROXY to Ensure Immutability and Availability

Once you maintain your GoLang dependencies as versioned modules, you can keep them as immutable entities by setting up a GOPROXY. In this way, you can always guarantee what a specific version of a module contains, so your builds are always repeatable.

Will you be using Go modules that are publicly available open-source packages? Ones you create and share with the OSS community? Modules you keep private to just your team? Here are ways to do each, or all at once, and ensure that those unchanging versioned modules are always available for your builds.

3. Use Artifactory Repository Layouts

When you build your app (using Go build), where will the binary artifacts generated be stored? Your build process can push those intermediate results to repositories in a binaries repository manager like Artifactory.

For these intermediate Go artifacts, you’ll use Artifactory’s generic repositories. Structuring those repositories can help you control the flow of your binaries through development, tests, and production with separate repositories for each of those stages. To help you do this, use the Custom Repository Layout feature of Artifactory with those generic repositories.

Create a custom layout that is similar to the following:

[org]/[name<.+>]/[module]-[arch<.+>]-[baseRev].[ext]

When you configure the custom layout, you should test the artifact path resolution to confirm how Artifactory will build module information from the path using the layout definitions.

4. Build Once and Promote

With your repositories properly set up for your Go builds, you can start to move them through your pipeline stages efficiently. 

Many software development procedures require a fresh complete or partial build at each staging transition of development, testing, and production. But as developers continue to change shared code, each new build introduces new uncertainties; you can’t be certain what’s in it. Even with safeguards to help assure deterministic builds that may still require repeating the same quality checks in each stage.  

Instead, build your Go-based microservices once, then promote them to the next stage once promotion criteria such as tests or scans are met. If you plan to containerize your Go microservice, the same principle applies: build each Docker image once and promote it through a series of staging repositories. In this way, you can guarantee that what was tested is exactly what is being released to production.

5. Avoid Monolithic Pipelines

Instead of a single, monolithic pipeline for your app, It’s better to have several, with each one building, testing, scanning, and promoting a different layer. This helps make your CI/CD process more flexible with different teams responsible for each layer, fostering a “fail fast” system that helps catch errors early.

For example, deploying a containerized application can typically be composed of five pipelines:

1. Build the Go application using the JFrog CLI. This pipeline pulls the source code, builds the application with Artifactory repositories for dependencies and output binaries, and tests and promotes from a dev repo to a staging repository in Artifactory.
2. Build a base layer of the containerized app, e.g. a Docker framework. Static or dynamic tags can be used based on company’s risk and upgrade policies. For example, If a dynamic tag such as alpine:3.10 is used as a base layer of Docker framework, then all patch updates will be included each time the Docker framework is built. This pipeline will include build, test and promote stages.
3. Pull the promoted artifacts produced by the prior pipelines and build a containerized app.This will also have build, test, scan and promote stages. 
4. Build a Helm chart that points to a statically tagged & promoted version of a containerized app that was produced by prior pipeline.
5. Deploy the containerized Go app to Kubernetes using the Helm chart.

Artifactory acts as your “source of truth” for your Go builds, providing a GOPROXY for both public and private modules and storing compiled binaries. Using the JFrog CLI to build your Go app helps the build process interact with Artifactory, and capture the build info that makes your builds fully traceable. Here is a sample snippet:

// Configure Artifactory
jfrog rt c
// Configure the project's repositories
jfrog rt go-config
// Build the project with go and resolve the project dependencies
from Artifactory.
jfrog rt go build --build-name=my-build --build-number=1
// Publish the package we build to Artifactory.
jfrog rt gp go v1.0.0 --build-name=my-build --build-number=1
// Collect environment variables and add them to the build info.
jfrog rt bce my-build 1
// Publish the build info to Artifactory.
jfrog rt bp my-build 1

To explore this more, take a look at our example demonstration files for GoLang CI/CD  that we’ll be using at GopherCon.