Pipeline as a Service: How To Test Pipelines in GitLab

When speaking about something-as-a-service, engineers should think about versioning and testing. Pipelines aren’t an exception. If some team wants to provide pipelines as a service they will face issues of testing new features during development and release. 

There are a lot of approaches for some development and testing software, but pipelines that describe CI/CD processes have to be capable of self-testing.

The article describes some approaches for testing pipelines that reduce time to development and delivery.

Prerequisite

There are at least two approaches to development and testing pipelines. With the first one development and testing are carried out in the app project and moving to the CI project files and release. Next, it’s necessary either to update all versions of pipelines to the application projects or to always use the latest version so that after moving, the new feature doesn't need to be updated using versions in the app projects. In this approach, a process of testing is carried out during development, and if something breaks down in a specific project after a new feature release, it’s necessary to use a stable version or to make some changes to the project(s).  

With the second one, development in the CI project happens immediately, but testing is carried out on the stub application (s) from the same repository. As in the previous case, the update version stage remains.

This approach requires that the development of a new feature always starts in the CI project, and it's important that stubs are always in the actual state.

In spite of the fact that these approaches work and can solve some development and testing pipeline issues, implementation limitations might cause difficulties during their application. 

Solutions

Pre-Alpha

So, there are two projects: devops/ci and apps/test. The first one contains CI files provided as a service. The base folder contains files that describe «what and how to do» e.g. build/test/deploy. 

Same-name folders in the parent directory contain files that describe «when to do» e.g if commit branch matches main.

devops/ci/
├── base
│   ├── build.yaml
│   ├── deploy.yaml
│   └── test.yaml
├── build
│   └── build.yaml
├── deploy
│   └── deploy.yaml
├── test
│   └── test.yaml
└── main.yaml

devops/ci/base/build.yaml

.build:
  stage: build
  script: |-
    echo build

devops/ci/build/build.yaml

include:
  local: base/build.yaml

build:
  extends: .build
  rules:
    - if: $CI_COMMIT_TAG

The second one plays the role of a test application.

apps/test/
└── .gitlab-ci.yml

apps/test/.gitlab-ci.yml

include:
  - project: devops/ci
    file: main.yaml
    ref: main

«ref: main» branch is used for simplification.

Alpha

In order to have an opportunity to test apps/test from devops/ci project let’s use «Downstream pipelines» in the GitLab.

Parent-Child Pipelines

A parent pipeline is a pipeline that triggers a downstream pipeline in the same project. The downstream pipeline is called a child pipeline.

Multi-Project Pipelines

A pipeline in one project can trigger downstream pipelines in another project, called multi-project pipelines. The user triggering the upstream pipeline must be able to start pipelines in the downstream project, otherwise the downstream pipeline fails to start.

devops/ci/.gitlab-ci.yml

apps_test:
  trigger:
    project: apps/test
    strategy: depend

Currently, if a new feature is merged to the default branch in the devops/ci project, it will be tested in the apps/test project automatically. The «strategy: depend» option indicates that the pipeline from the devops/ci project will be completed successfully if the pipeline from apps/test is completed without error.

This functionality isn’t really useful for understandable reasons since all errors must be fixed on the development stage in the feature branch and not post-fact.

Beta

In order to improve the approach provided in the «Alpha» version, it’s necessary to learn somehow to control downstream pipelines and, especially while triggering it, try to somehow pass feature branch names. According to «Multi-project pipelines»

Are triggered from another project’s pipeline, but the upstream (triggering)pipeline does not have much control over the downstream (triggered) pipeline. However, it can choose the ref of the downstream pipeline, and pass CI/CD variables to it.

Excellent! That’s exactly what we need! According to «pass a predefined variable», the CI_COMMIT_REF_NAME variable value is necessary to save in the other variable.

According to «predefined CI/CD variables reference» variable. During a triggering downstream pipeline, environment variables will be passed to the child pipeline, which gives an opportunity to develop and test new features in the feature branch devops/ci project until merging in the default branch.

For more details, please see: «Pass a predefined variable».

First, it’s necessary to make changes from «ref: main» to «ref: $PP__CI_COMMIT_REF_NAME» in the default branch apps/test project, but it’s necessary to create a feature branch in the DevOps/ci project and start to test right now! 

apps/test/.gitlab-ci.yml

 include:
  - project: devops/ci
    file: main.yaml
    ref: $PP__CI_COMMIT_REF_NAME

devops/ci/.gitlab-ci.yml

variables:
  PP__CI_COMMIT_REF_NAME: $CI_COMMIT_REF_NAME

apps_test:
  trigger:
    project: apps/test
    strategy: depend

It isn’t possible to use the CI_COMMIT_BRANCH variable because it isn’t defined in events, such as: MergeRequest and Tag.

Prefix PP__(two underline) means «pipeline» and describes local variables or devops/ci projects.

Release Candidate

Although the current implementation solves the approach limitation described in the «Prerequisite» section, there is one more issue that makes this approach user-unfriendly.

If we try to run from apps/test pipeline directly, it will not run because .gitlab-ci.yml will be invalid. This puts limitations on the usage of this approach, and unfortunately, it’s not possible to define «variables.PP__CI_COMMIT_REF_NAME: main» because variables don't exist while the included is being resolved.

There are four solutions:

1. Run a pipeline manually and define variables via the UI
2. Use push options
3. Define variables as instance/group/project level
4. Use dirty hack via YAML anchors in .gitlab-ci.yml

The first and second options can’t be used for understandable reasons, but the fourth one works only via explicitly set YAML anchors in the target project. 

For more details, please see the issue.

According to «CI/CD variable precedence», I propose to use the third option because this gives the opportunity to centrally control the version via instance/group variables and, if necessary, override the version via explicit definition in the target project. We can do it via project-level variables or dirty hack YAML anchors.

For more details, please see «Override a defined CI/CD variable».

Release

In fact, testing for pipelines is ready, but at least there are several issues that should be solved before the release:

• Unlike the stubs option mostly the result of build in the target pipelines is an artifact
• Pipelines in the target projects are launched according to particular rules
• Every pipeline in the target project that has to be tested must be defined in the CI project

In order to make unification for all three issues it’s necessary to define one more variable, which will indicate that a pipeline has been launched from the CI project.

Depending on the situation, artifacts either need to be saved with a specific name or skip this step.

For example, if add to devops/ci/base/build.yaml:

if [ ${PP__IS_ENABLED} ]; then
  echo “Artifact will be skipped”
else
  echo “Artifact will be saved”
fi

An artifact will not be created if PP__IS_ENABLED=true.

Similarly, this issue can be solved by launching rules of testing pipelines in the target projects from devops/ci. If so, add the following rule for each job:

rules:
  - if: $CI_PIPELINE_SOURCE == 'pipeline' && $PP__IS_ENABLED

Then, all the pipelines can be launched successfully for testing, and this will not break the logic of general workflow.

In a perfect world, the number of testing pipelines must be equal to the number of projects with enabled CI/CD. So, we can conclude that manual addition will become hell when there are a lot of projects. In order to solve the issue, it’s necessary to use «Dynamic child pipelines».  Dynamic pipelines allow you to generate the CI file and, in the next step, trigger it as a child pipeline.

Workflow will be the following:

1. Get all namespaces
2. Get all groups/subgroups in the namespaces
3. Get all projects in the groups/subgroups
4. Check activated CI/CD, existing branches, some variables, etc.
5. Check the pipeline file in the project
6. Generate CI file

The PoC script that implements the workflow above was published in my GitHub project. This script uses API GitLab to prepare a list of projects for testing. According to the workflow above, there are several conditions that are applied for each project, and one of them is the existence of the PP__IS_ENABLED project level variable. If the variable is defined (i.e., exists), this project will be added to the list of projects which jobs are generated from.

The script requires personal access tokens with permission, such as read_api, read_user, and read_repository, for launching. If projects have been configured with «Allow access to your project with a job token» a personal access token isn’t required, and it’s possible to use CI_JOB_TOKEN.

For more details, please see:

• Limit job token scope for public or internal projects
• Disable the job token scope allowlist

devops/ci/.gtilab-ci.yml

image: python:3.12.1-alpine3.18
generate-pipeline:
  stage: build
  script: |-
    echo -e "\e[0Ksection_start:`date +%s`:my_first_section[collapsed=true]\r\e[0KPrepare environment"
    apk add --no-cache py-pip
    pip install --upgrade --no-cache pipenv
    pipenv install
    echo -e "\e[0Ksection_end:`date +%s`:my_first_section\r\e[0K"
    pipenv run python .gitlab-ci.py
  artifacts:
    paths:
      - generated-config.yml

test:
  stage: test
  trigger:
    include:
      - artifact: generated-config.yml
        job: generate-pipeline
    strategy: depend
  needs:
    - generate-pipeline

Generated File Example

stages:
- test
apps-test-test:
  trigger:
    project: apps/test
    strategy: depend
  stage: test
  variables:
    PP__IS_ENABLED: 'true'
    PP__CI_COMMIT_REF_NAME: $CI_COMMIT_REF_NAME

Conclusion

Although the approach described in the article solves approach limitations from «Prerequisite» section. It also has the following pros/cons:

Pros

• New feature implementation in the CI project via testing in the target projects
• The opportunity for a smooth update via dynamic generation of the testing pipelines list
• Stubs application rejection 

Cons

• When the result of the build is artifact processing is required  
• Some variables require to be controlled via group/instance level

The example is not for using as-is but it must inspire the next step towards improvement of the approach of testing pipelines.

All diagrams were created from code.

References

• Pipeline-as-a-Service: PoC
• python-gitlab
• python-gitlab-ci