GitLab CI runner can be contained in a completely rootless environment. It can start as a non-root user, and work with a rootless Podman instance as a Docker runner. And here is how I achieved it.
My CI host configuration:
- Ubuntu 20.04
- Podman 3.4.2
- GitLab Runner 14.5.0
A similar procedure can be applied to other distros as well.
Install Podman
Use the following script for installing the latest version of Podman onto a Ubuntu.
|
1 2 3 4 5 6 |
. /etc/os-release echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list curl -L "https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key" | sudo apt-key add - sudo apt-get update sudo apt-get -y upgrade sudo apt-get -y install podman |
For other distros, please refer to the official documentation.
Install GitLab Runner
|
1 2 3 4 5 6 |
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash sudo apt-get install gitlab-runner strace chown -R root:gitlab-runner /etc/gitlab-runner chmod 750 /etc/gitlab-runner chmod 640 /etc/gitlab-runner/config.toml |
For other distros, please refer to the official documentation.
Note that we installed strace here, which we will be using later.
Config User Namespace Mapping
You need to allocate a range of UID/GIDs per user. You might adjust them for your needs.
|
1 2 3 4 5 6 |
sudo usermod --add-subuids 200000-265536 --add-subgids 200000-265536 gitlab-runner cat > /etc/sysctl.d/90-podman.conf <<EOF kernel.unprivileged_userns_clone=1 EOF sysctl -f /etc/sysctl.d/90-podman.conf |
Enable Podman Service
|
1 2 3 4 5 6 7 |
loginctl enable-linger gitlab-runner mkdir -p ~gitlab-runner/.config/systemd/user/sockets.target.wants ~gitlab-runner/.config/systemd/user/multi-user.target.wants chown gitlab-runner:gitlab-runner -R ~gitlab-runner/.config/systemd/user/* XDG_RUNTIME_DIR="/run/user/$(id -u gitlab-runner)" sudo -Eu gitlab-runner -- systemctl --user daemon-reload XDG_RUNTIME_DIR="/run/user/$(id -u gitlab-runner)" sudo -Eu gitlab-runner -- systemctl --user enable --now podman.socket podman.service sudo -u gitlab-runner -- podman network create podman |
Then we need to verify if Podman is enabled correctly:
|
1 2 |
sudo su - gitlab-runner bash -c 'podman system info' sudo su - gitlab-runner bash -c 'podman run --rm hello-world' |
Enable GitLab Runner Service
|
1 2 3 4 5 6 7 8 9 10 11 12 |
systemctl stop gitlab-runner mkdir -p /etc/systemd/system/gitlab-runner.service.d cat > /etc/systemd/system/gitlab-runner.service.d/override.conf <<EOF [Service] User=gitlab-runner Group=gitlab-runner ExecStart= ExecStart=strace -e getuid,getgid -e inject=getuid:retval=0 -e inject=getgid:retval=0 -- /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner EOF systemctl daemon-reload systemctl enable --now gitlab-runner |
Run gitlab-runner register as usual. After registration, set runners[].docker.host = "unix:///run/user/996/podman/podman.sock" , then restart the runner service. ( 996 is the UID for gitlab-runner user. It might differ in your environment. )
Note: the weird strace command tricks GitLab runner into thinking it is started by the root user, so it launches into system mode. By default, the runner will be started in user mode which is not helpful in our use case.
Known Issues
CentOS 6 containers does not work
If you need to run CentOS 6 or other similar old Docker containers, you need to append vsyscall=emulate to the kernel command line and reboot.
Not able to pull container using short names
|
1 2 3 |
cat > /etc/containers/registries.conf <<EOF unqualified-search-registries = ["docker.io"] EOF |
Using Self-signed CA
For GitLab runner to connect to an HTTPS endpoint, you only need to install the CA to the system-level certificate storage, then change runners[].url to a HTTPS URL.
For Git to successfully pull repos using HTTPS, you have to first modify the official helper images, then set runners[].docker.helper_image = "my.registry.local/gitlab/gitlab-runner-helper:x86_64-v${CI_RUNNER_VERSION}" . If you really don’t care about security, set runners[].docker.environment = ["GIT_SSL_NO_VERIFY=true"] (very insecure, very dangerous).
For the actual CI workload to connect to HTTPS sites with self-signed CA, set runners[].docker.volumes = ["/cache", "/usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro", "/etc/ssl/certs:/etc/ssl/certs:ro"] . This might not work if your container or host OS doesn’t follow the conventions.
Changed subuid/subgid mapping, need to reset podman
If you changed subuid/subgid mapping or did not set it up correctly initially, Podman might act weird. Run the following commands to resolve (and you need to manually fix any others errors displayed when running these commands). Note the commands will completely wipe every bit of data (container, image, volumes) stored in your Podman database.
|
1 2 3 4 5 6 |
USERNAME="gitlab-runner" USER_UID=$(id -u "$USERNAME") rm -rf ~"$USERNAME"/.{config,local/share}/containers /run/user/"$USER_UID"/{libpod,runc,vfs-*} su - "$USERNAME" systemctl --user restart dbus su - "$USERNAME" podman system reset su - "$USERNAME" podman system migrate |
Unable to upgrade to tcp, received 409
You might receive the following error:
|
1 |
ERROR: Job failed (system failure): prepare environment: unable to upgrade to tcp, received 409 (exec.go:59:0s). Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information |
Check your file permissions. Try disable the cache configuration on the runner and try again. The outermost cache directory might need a permission of 0755 .
CNI network “podman” not found
Run sudo -u gitlab-runner -- podman network create podman .
Generic Debugging Advices
To enable verbose mode on the GitLab runner, you can either add log_level = "debug" on the top level of the config file, or use gitlab-runner --debug run -c /etc/gitlab-runner/config.toml to launch a temporary runner.
To debug Docker API errors, you can use BPF to capture traffic on the UNIX socket, or convert the socket to a TCP listener with socat TCP-LISTEN:12345,reuseaddr,fork UNIX-CLIENT:/run/user/996/podman/podman.sock , use host = "tcp://127.0.0.1:12345" in the runner config to connect, then use tcpdump -i lo -w docker.pcap tcp port 12345 to see what happened.
Bonus: Use tmpfs to run the containers
This will clear all your previous podman data, so make sure you back up important images and containers.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
mkdir -p /mnt/gitlab-runner-volatile echo "tmpfs /mnt/gitlab-runner-volatile tmpfs defaults,noatime,size=64g,uid=996,gid=996,mode=0755,huge=always" >> /etc/fstab mount /mnt/gitlab-runner-volatile su - gitlab-runner podman system reset cat > ~gitlab-runner/.config/containers/storage.conf <<EOF [storage] driver = "overlay" runroot = "/mnt/gitlab-runner-volatile/podman_containers/runroot" graphroot = "/mnt/gitlab-runner-volatile/podman_containers/graphroot" rootless_storage_path = "/mnt/gitlab-runner-volatile/podman_containers/storage" [storage.options.overlay] mountopt = "nodev,metacopy=on" mount_program = "/usr/bin/fuse-overlayfs" EOF XDG_RUNTIME_DIR=/run/user/$(id -u gitlab-runner) systemctl --user restart podman.service |
Additional configuration required for SELinux-enabled hosts.
References
- 使用 CONTAINER-TOOLS API
- Why doesn’t my systemd user unit start at boot?
- rootless service not work with systemd socket activation
- Add support for Podman to GitLab Runner
- Configuring container networking with Podman
- How to expose a UNIX domain socket directly over TCP
- Can’t use tmpfs as runroot for containers