Python SDKs Package Management in GCP Artifact Registry

Introduction

Using a centralized, private repository to host SDK as a package not only enables code reuse but also simplifies and secures the existing software delivery pipeline. By using the same formats and tools as you would in the open-source ecosystem, you can leverage the same advantages, simplify building, and keep business logic and applications secure.

Storing SDK packages in Google Cloud Artifact Registry not only enables SDK code reuse but also simplifies and secures your existing build pipeline. In addition to bringing your internal packages to a managed repository, using Artifact Registry also allows you to take additional steps to improve the security of your software delivery pipeline. 

In this tutorial, we will learn how to:

1. Set up an Artifact Registry for Python SDK repository
2. Upload an SDK package
3. Install the SDK package

1. Set Up an Artifact Registry for Python SDK Repository

As a part of setting up an Artifact Registry for Python SDK repository, let's first set up a virtual environment called "pythnsdk" and then install Twine and the Artifact Registry keyring backend in the virtual environment.

python3 -m venv pythnsdk
source pythnsdk/bin/activate
python -m pip install -U pip
pip install twine keyrings.google-artifactregistry-auth
keyring --list-backends

Log in to Google Cloud CLI with service account credentials.

gcloud auth activate-service-account [SERVICE_ACCOUNT] --key-file=/path/key.json --project=PROJECT_ID

Now, create a new Python package repository in the current project named "python-sdk-repo" in the location us-west1.

gcloud artifacts repositories create python-sdk-repo \
--repository-format=python \
--location=us-west1 \
--description="Python package repository"

To set up authentication with the Artifact Registry keyring backend, run the following command to print the repository configuration to add to your Python project.

gcloud artifacts print-settings python

Update the .pypirc file that specifies repositories for publishing Python packages. Verify $HOME/.pypirc and $VIRTUAL_ENV/pip.conf files and check whether repository settings are matching to the intended Artifact Registry repository and other GCP project details that were set up earlier.

cat $HOME/.pypirc

[distutils]
index-servers =
python-sdk-repo

[python-sdk-repo]
repository: https://us-west11-python.pkg.dev/PROJECT_ID/python-sdk-repo/

cat $VIRTUAL_ENV/pip.conf

[global]
extra-index-url = https://us-west1-python.pkg.dev/PROJECT_ID/python-sdk-repo/simple/

mkdir python-sdk

cd python-sdk

Now download the whole Python SDK folder structure from the project Git repository (this folder structure contains a Python SDK that is already packaged in dist/ folder ) to the python-sdk folder

2. Upload an SDK package

Upload the packages to the Artifact Registry Python SDK repository from your dist directory.

twine upload --repository-url https://us-west1-python.pkg.dev/PROJECT_ID/python-sdk-repo/ dist/*

Now view the package in the repository.

gcloud artifacts packages list --repository=python-sdk-repo

To view versions of a package, run the following command:

gcloud artifacts versions list --package=PACKAGE

Where PACKAGE is the package ID.

3. Install the SDK package.

Run the following command to install the Python SDK package uploaded to Artifact Registry in local.

pip install --index-url https://us-west1-python.pkg.dev/PROJECT_ID/python-sdk-repo/simple/ projectsdk

Note: Use Cloud Identity and Access Management (IAM) for repository access control.

The above procedure has been successfully implemented and tested on the GCP client environment, verified in the below screenshot on what messages are seen after successful installation of a SDK package as reference.