CI/CD Pipeline Configuration

GitLab CI/CD build for a repository are controlled placing a file named .gitlab_ci.yml in the root folder of the repository. The expected content of the file is specified on GitLab's gitlab_ci.yml Documentation page.

GitLab Runner

Installing

Follow the instructions on this GitLab documentation page, summarized below, where you set up a user account for GitLab Runner and make it launch automatically when your system starts.

sudo curl -L --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64"
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

You can check the status of the service by typing

sudo gitlab-runner status

Using with Docker

If GitLab Runner is to build your software inside of Docker containers, it needs to able to issue Docker commands. Therefore, make sure Docker is installed and the user gitlab-runner is a member of the docker group:

sudo usermod -aG docker gitlab-runner

which, in virtualized Linux environments, may require a reboot to take effect.

The integration of GitLab Runner with Docker goes deeper than just instantiating a container: Before launching the build scripts from .gitlab_ci.yml, GitLab Runner will clone the repository into a build folder inside the container! The default path is /build/ followed by the project path on the GitLab server. The build scripts simply leave it to GitLab Runner to provide them with the correct commit from the correct branch.

Registering the Runner

After GitLab Runner is installed, you will need to register the runner with the GitLab code repository. Typing

sudo gitlab-runner register

launches the interactive registration, which will prompt you for each of the configuration parameters. You can also supply some or all of the parameters directly in the command:

sudo gitlab-runner register \
  --url "https://" \
  --registration-token <secret token> \
  --description "docker-swift:5.4" \
  --executor "docker" \
  --docker-image swift:5.4

where <secret token> is the token obtained from GitLab project settings → CI/CD → Runners → Specific Runners.

Registration finishes by generating the /etc/gitlab-runner/config.toml file similar to this:

concurrent = 1 
check_interval = 0 

[session_server]
  session_timeout = 1800

[[runners]]
  name = "ubuntu-at-home"
  url = "https://gitlab.com/"
  token = <secret token>
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "keytree"
    privileged = false
    disable_entrypoint_overwrite = false
    pull_policy = "never"
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

after which, the gitlab-runner service will notice that the configuration file has changed and applies the new configuration.

Note that the pull_policy option was explicitly set to "never" instead of going with the default "always". This prevents GitLab Runner from searching for the specified Docker image on Docker Hub and directs the query to the local docker engine!

Troubleshooting

Some CI/CD pipelines will fail due to the presence of clear_console in /home/gitlab-runner/.bash_logout. Delete or comment out the contents of that file.

How to create a Product Webpage

How to create Product Releases with Binaries

The best way to add binaries to project releases on GitLab, is to create a job which uploads the release assets as a generic package, then creates a release by referring to the uploaded assets.

In the project root folder, add a .gitlab-ci.yml file with the contents based on the example below. Note that the upload script makes use of **CI_JOB_TOKEN**, which GitLab passes to the build runner for use in build scripts that need to authenticate themselves when sending files to GitLab.

stages:
  - upload
  - release
  
variables:
  PRODUCT_NAME: "MyProject"
  ASSET_FILE_NAME: "${PRODUCT_NAME}-${CI_COMMIT_TAG}.dmg"
  ASSET_FILE_SOURCE_PATH: "${HOME}/binary_products/${ASSET_FILE_NAME}"
  PACKAGE_REGISTRY_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${PRODUCT_NAME}/${CI_COMMIT_TAG}/"

upload:
  stage: upload
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - |
      curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file ${ASSET_FILE_SOURCE_PATH} ${PACKAGE_REGISTRY_URL}/${ASSET_FILE_NAME}

release:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:v0.4.0
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - |
      /usr/local/bin/release-cli create --name "Release $CI_COMMIT_TAG" --tag-name $CI_COMMIT_TAG \
        --assets-link "{\"name\":\"${ASSET_FILE_NAME}\",\"url\":\"${PACKAGE_REGISTRY_URL}${ASSET_FILE_NAME}\"}"

where release-cli is a tool (written in Go) that instructs GitLab to create a release for the given Git tag and with the given binary assets.

In VirtualBox/Ubuntu register a GitLab Runner instance for the project. This runner will not build the product. It will only be used to upload the binary assets.
In VirtualBox/Ubuntu install release-cli by downloading from here and running

sudo apt install golang
cd release-cli-v0.8.0
go build cmd/release-cli/main.go
cp main /usr/local/bin/release-cli

Create the binary to be released and place it in the folder expected by the upload script. You may want to set up a VirtualBox shared folder.
In GitLab, create a tag for the version to be released. This will trigger the execution of the upload job in GitLab Runner.
To refer the user to a download page containing all releases, use the following URL:

https://gitlab.com/<project group name>/<project name>/-/releases