GitLab CI

In addition to being a Git repository service, GitLab provides Continuous Integration features.

CI Pipeline Configuration

A GitLab CI pipeline for a repository is established by placing a file named .gitlab_ci.yml in the root folder of the repository. The contents of the file specify the structure and the actions of the pipeline. See GitLab's gitlab_ci.yml documentation page to see what's possible.

GitLab Runner

GitLab Runner is GitLab's own solution for a background service that can be hosted separately from the GitLab server and is responsible for executing the CI pipeline.

Installing

Follow the instructions on this GitLab documentation page, summarized below, where you set up a user account for GitLab Runner and make it launch automatically when your system starts.

sudo curl -L --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64"
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

You can check the status of the service by typing

sudo gitlab-runner status

Using with Docker

If GitLab Runner is to build your software inside of Docker containers, it needs to able to issue Docker commands. Therefore, make sure Docker is installed and the user gitlab-runner is a member of the docker group:

sudo usermod -aG docker gitlab-runner

which, in virtualized Linux environments, may require a reboot to take effect.

The integration of GitLab Runner with Docker goes deeper than just instantiating a container: Before launching the build scripts from .gitlab_ci.yml, GitLab Runner will clone the repository into a build folder inside the container! The default path is /build/ followed by the project path on the GitLab server. The build scripts simply leave it to GitLab Runner to provide them with the correct commit from the correct branch.

Registering the Runner

After GitLab Runner is installed, you will need to register the runner with the GitLab code repository. Typing

sudo gitlab-runner register

launches the interactive registration, which will prompt you for each of the configuration parameters. You can also supply some or all of the parameters directly in the command:

sudo gitlab-runner register \
  --url "https://" \
  --registration-token <secret token> \
  --description "docker-swift:5.4" \
  --executor "docker" \
  --docker-image swift:5.4

where <secret token> is the token obtained from GitLab project settings → CI/CD → Runners → Specific Runners.

Registration finishes by generating the /etc/gitlab-runner/config.toml file similar to this:

concurrent = 1 
check_interval = 0 

[session_server]
  session_timeout = 1800

[[runners]]
  name = "ubuntu-at-home"
  url = "https://gitlab.com/"
  token = <secret token>
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "keytree"
    privileged = false
    disable_entrypoint_overwrite = false
    pull_policy = "never"
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0 

after which, the gitlab-runner service will notice that the configuration file has changed and applies the new configuration.

Note that the pull_policy option was explicitly set to "never" instead of going with the default "always". This prevents GitLab Runner from searching for the specified Docker image on Docker Hub and directs the query to the local docker engine!

Troubleshooting

• Some CI/CD pipelines will fail due to the presence of clear_console in /home/gitlab-runner/.bash_logout. Delete or comment out the contents of that file.

How to create Product Releases with Binaries

In GitLab, a release can be created from a tagged version of the repository. Binary assets (for example, a pre-built and packaged product) can be made part of a release after uploading the assets into the generic packages repository, which is separate from the project sources repository. Uploading into the generic packages repository requires special permissions for which you would normally generate special access tokens (which can be used in a curl command). However, the most convenient way of obtaining permissions from the GitLab server for uploading binary assets, is during the execution of a CI job, where a special access token is handed to the CI runner from the GitLab server. After uploading the asset, the same CI runner will then request the creation of a release that includes the uploaded asset.

• In the project root folder, add a .gitlab-ci.yml file with the contents based on the example below. Note that the upload script makes use of **CI_JOB_TOKEN**, which GitLab passes to the build runner for use in build scripts that need to authenticate themselves when sending files to GitLab.
  

stages:
  - upload
  - release
  
variables:
  PRODUCT_NAME: "MyProject"
  ASSET_FILE_NAME: "${PRODUCT_NAME}-${CI_COMMIT_TAG}.dmg"
  ASSET_FILE_SOURCE_PATH: "${HOME}/binary_products/${ASSET_FILE_NAME}"
  PACKAGE_REGISTRY_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${PRODUCT_NAME}/${CI_COMMIT_TAG}/"

upload:
  stage: upload
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - |
      curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file ${ASSET_FILE_SOURCE_PATH} ${PACKAGE_REGISTRY_URL}/${ASSET_FILE_NAME}

release:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:v0.4.0
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - |
      /usr/local/bin/release-cli create --name "Release $CI_COMMIT_TAG" --tag-name $CI_COMMIT_TAG \
        --assets-link "{\"name\":\"${ASSET_FILE_NAME}\",\"url\":\"${PACKAGE_REGISTRY_URL}${ASSET_FILE_NAME}\"}"

where release-cli is a tool (written in Go) that instructs GitLab to create a release for the given Git tag and with the given binary assets.

• In VirtualBox/Ubuntu register a GitLab Runner instance for the project. This runner will not build the product. It will only be used to upload the binary assets.
• In VirtualBox/Ubuntu install release-cli by downloading from here and running

sudo apt install golang
cd release-cli-v0.8.0
go build cmd/release-cli/main.go
cp main /usr/local/bin/release-cli

• Create the binary to be released and place it in the folder expected by the upload script. When using a VirtualBox shared folder, make sure the user gitlab-runner is in the vboxsf group.

sudo usermod -a -G vboxsf gitlab-runner

• In GitLab, create a tag for the version to be released. This will trigger the execution of the upload job in GitLab Runner.
• The download page containing all releases will be at

https://gitlab.com/<project group name>/<project name>/-/releases