GitLab CI Runner on Rootless Podman

GitLab CI runner can be contained in a completely rootless environment. It can start as a non-root user, and work with a rootless Podman instance as a Docker runner. And here is how I achieved it.

My CI host configuration:

  • Ubuntu 20.04
  • Podman 3.4.2
  • GitLab Runner 14.5.0

A similar procedure can be applied to other distros as well.

Install Podman

Use the following script for installing the latest version of Podman onto a Ubuntu.

For other distros, please refer to the official documentation.

Install GitLab Runner

For other distros, please refer to the official documentation.

Note that we installed strace here, which we will be using later.

Config User Namespace Mapping

You need to allocate a range of UID/GIDs per user. You might adjust them for your needs.

Enable Podman Service

Then we need to verify if Podman is enabled correctly:

Enable GitLab Runner Service

Run gitlab-runner register as usual. After registration, set runners[].docker.host = "unix:////run/user/996/podman.sock" , then restart the runner service. ( 996  is the UID for gitlab-runner user. It might differ in your environment. )

Note: the weird strace command tricks GitLab runner into thinking it is started by the root user, so it launches into system mode. By default, the runner will be started in user mode which is not helpful in our use case.

Known Issues

CentOS 6 containers does not work

If you need to run CentOS 6 or other similar old Docker containers, you need to append vsyscall=emulate  to the kernel command line and reboot.

Not able to pull container using short names

Using Self-signed CA

For GitLab runner to connect to an HTTPS endpoint, you only need to install the CA to the system-level certificate storage, then change runners[].url to a HTTPS URL.

For Git to successfully pull repos using HTTPS, you have to first modify the official helper images, then set runners[].docker.helper_image = "my.registry.local/gitlab/gitlab-runner-helper:x86_64-v${CI_RUNNER_VERSION}" . If you really don’t care about security, set  runners[].docker.environment = ["GIT_SSL_NO_VERIFY=true"] (very insecure, very dangerous).

For the actual CI workload to connect to HTTPS sites with self-signed CA, set runners[].docker.volumes = ["/cache", "/usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro", "/etc/ssl/certs:/etc/ssl/certs:ro"] . This might not work if your container or host OS doesn’t follow the conventions.

Changed subuid/subgid mapping, need to reset podman

If you changed subuid/subgid mapping or did not set it up correctly initially, Podman might act weird. Run the following commands to resolve (and you need to manually fix any others errors displayed when running these commands). Note the commands will completely wipe every bit of data (container, image, volumes) stored in your Podman database.

Unable to upgrade to tcp, received 409

You might receive the following error:

Check your file permissions. Try disable the cache configuration on the runner and try again. The outermost cache directory might need a permission of 0755 .

CNI network “podman” not found

Run sudo -u gitlab-runner -- podman network create podman .

Generic Debugging Advices

To enable verbose mode on the GitLab runner, you can either add log_level = "debug" on the top level of the config file, or use gitlab-runner --debug run -c /etc/gitlab-runner/config.toml to launch a temporary runner.

To debug Docker API errors, you can use BPF to capture traffic on the UNIX socket, or convert the socket to a TCP listener with socat TCP-LISTEN:12345,reuseaddr,fork UNIX-CLIENT:/run/user/996/podman/podman.sock , use host = "tcp://127.0.0.1:12345" in the runner config to connect, then use tcpdump -i lo -w docker.pcap tcp port 12345 to see what happened.

Bonus: Use tmpfs to run the containers

This will clear all your previous podman data, so make sure you back up important images and containers.

Additional configuration required for SELinux-enabled hosts.


References

Leave a Reply

Your email address will not be published. Required fields are marked *