GitLab CI runner can be contained in a completely rootless environment. It can start as a non-root user, and work with a rootless Podman instance as a Docker runner. And here is how I achieved it.<\/p>\n
My CI host configuration:<\/p>\n
A similar procedure can be applied to other distros as well.<\/p>\n
<\/p>\n
Use the following script for installing the latest version of Podman onto a Ubuntu.<\/p>\n
. \/etc\/os-release\r\necho \"deb https:\/\/download.opensuse.org\/repositories\/devel:\/kubic:\/libcontainers:\/stable\/xUbuntu_${VERSION_ID}\/ \/\" | sudo tee \/etc\/apt\/sources.list.d\/devel:kubic:libcontainers:stable.list\r\ncurl -L \"https:\/\/download.opensuse.org\/repositories\/devel:\/kubic:\/libcontainers:\/stable\/xUbuntu_${VERSION_ID}\/Release.key\" | sudo apt-key add -\r\nsudo apt-get update\r\nsudo apt-get -y upgrade\r\nsudo apt-get -y install podman<\/pre>\nFor other distros, please refer to the official documentation<\/a>.<\/p>\nInstall GitLab Runner<\/h2>\ncurl -L \"https:\/\/packages.gitlab.com\/install\/repositories\/runner\/gitlab-runner\/script.deb.sh\" | sudo bash\r\nsudo apt-get install gitlab-runner strace\r\n\r\nchown -R root:gitlab-runner \/etc\/gitlab-runner\r\nchmod 750 \/etc\/gitlab-runner\r\nchmod 640 \/etc\/gitlab-runner\/config.toml<\/pre>\nFor other distros, please refer to the official documentation<\/a>.<\/p>\nNote that we installed strace here, which we will be using later.<\/p>\n
Config User Namespace Mapping<\/h2>\n
You need to allocate a range of UID\/GIDs per user. You might adjust them for your needs.<\/p>\n
sudo usermod --add-subuids 200000-265536 --add-subgids 200000-265536 gitlab-runner\r\n\r\ncat > \/etc\/sysctl.d\/90-podman.conf <<EOF\r\nkernel.unprivileged_userns_clone=1\r\nEOF\r\nsysctl -f \/etc\/sysctl.d\/90-podman.conf<\/pre>\nEnable Podman Service<\/h2>\nloginctl enable-linger gitlab-runner\r\n\r\nmkdir -p ~gitlab-runner\/.config\/systemd\/user\/sockets.target.wants ~gitlab-runner\/.config\/systemd\/user\/multi-user.target.wants\r\nchown gitlab-runner:gitlab-runner -R ~gitlab-runner\/.config\/systemd\/user\/*\r\nXDG_RUNTIME_DIR=\"\/run\/user\/$(id -u gitlab-runner)\" sudo -Eu gitlab-runner -- systemctl --user daemon-reload\r\nXDG_RUNTIME_DIR=\"\/run\/user\/$(id -u gitlab-runner)\" sudo -Eu gitlab-runner -- systemctl --user enable --now podman.socket podman.service\r\nsudo -u gitlab-runner -- podman network create podman<\/pre>\nThen we need to verify if Podman is enabled correctly:<\/p>\n
sudo su - gitlab-runner bash -c 'podman system info'\r\nsudo su - gitlab-runner bash -c 'podman run --rm hello-world'<\/pre>\nEnable GitLab Runner Service<\/h2>\nsystemctl stop gitlab-runner\r\nmkdir -p \/etc\/systemd\/system\/gitlab-runner.service.d\r\ncat > \/etc\/systemd\/system\/gitlab-runner.service.d\/override.conf <<EOF\r\n[Service]\r\nUser=gitlab-runner\r\nGroup=gitlab-runner\r\nExecStart=\r\nExecStart=strace -e getuid,getgid -e inject=getuid:retval=0 -e inject=getgid:retval=0 -- \/usr\/bin\/gitlab-runner run --working-directory \/home\/gitlab-runner --config \/etc\/gitlab-runner\/config.toml --service gitlab-runner --user gitlab-runner\r\n\r\nEOF\r\nsystemctl daemon-reload\r\nsystemctl enable --now gitlab-runner<\/pre>\nRun gitlab-runner register<\/span> as usual. After registration, set runners[].docker.host = “unix:\/\/\/run\/user\/996\/podman\/podman.sock”<\/span> , then restart the runner service. (996<\/span>\u00a0 is the UID for gitlab-runner<\/span> user. It might differ in your environment. )<\/p>\nNote: the weird strace<\/span> command tricks GitLab runner into thinking it is started by the root user<\/a>, so it launches into system mode. By default, the runner will be started in user mode which is not helpful in our use case<\/a>.<\/p>\nKnown Issues<\/h1>\nCentOS 6 containers does not work<\/h2>\n
If you need to run CentOS 6 or other similar old Docker containers, you need to append vsyscall=emulate<\/span>\u00a0 to the kernel command line and reboot.<\/p>\nNot able to pull container using short names<\/h2>\ncat > \/etc\/containers\/registries.conf <<EOF\r\nunqualified-search-registries = [\"docker.io\"]\r\nEOF<\/pre>\nUsing Self-signed CA<\/h2>\n
For GitLab runner to connect to an HTTPS endpoint, you only need to install the CA to the system-level certificate storage, then change runners[].url<\/span> to a HTTPS URL.<\/p>\nFor Git to successfully pull repos using HTTPS, you have to first modify the official helper images<\/a>, then set runners[].docker.helper_image = “my.registry.local\/gitlab\/gitlab-runner-helper:x86_64-v${CI_RUNNER_VERSION}”<\/span> . If you really don’t care about security, set\u00a0 runners[].docker.environment = [“GIT_SSL_NO_VERIFY=true”]<\/span> (very insecure, very dangerous).<\/p>\nFor the actual CI workload to connect to HTTPS sites with self-signed CA, set runners[].docker.volumes = [“\/cache”, “\/usr\/local\/share\/ca-certificates:\/usr\/local\/share\/ca-certificates:ro”, “\/etc\/ssl\/certs:\/etc\/ssl\/certs:ro”]<\/span> . This might not work if your container or host OS doesn’t follow the conventions.<\/p>\nChanged subuid\/subgid mapping, need to reset podman<\/h2>\n
If you changed subuid\/subgid mapping or did not set it up correctly initially, Podman might act weird. Run the following commands to resolve (and you need to manually fix any others errors displayed when running these commands). Note the commands will completely wipe every bit of data (container, image, volumes) stored in your Podman database.<\/p>\n
USERNAME=\"gitlab-runner\"\r\nUSER_UID=$(id -u \"$USERNAME\")\r\nrm -rf ~\"$USERNAME\"\/.{config,local\/share}\/containers \/run\/user\/\"$USER_UID\"\/{libpod,runc,vfs-*}\r\nsu - \"$USERNAME\" systemctl --user restart dbus\r\nsu - \"$USERNAME\" podman system reset\r\nsu - \"$USERNAME\" podman system migrate<\/pre>\nUnable to upgrade to tcp, received 409<\/h2>\n
You might receive the following error:<\/p>\n
ERROR: Job failed (system failure): prepare environment: unable to upgrade to tcp, received 409 (exec.go:59:0s). Check https:\/\/docs.gitlab.com\/runner\/shells\/index.html#shell-profile-loading for more information<\/pre>\nCheck your file permissions. Try disable the cache configuration on the runner and try again. The outermost cache directory might need a permission of 0755<\/span>\u00a0.<\/p>\nCNI network “podman” not found<\/h2>\n
Run sudo -u gitlab-runner — podman network create podman<\/span> .<\/p>\nGeneric Debugging Advices<\/h2>\n
To enable verbose mode on the GitLab runner, you can either add log_level = “debug”<\/span> on the top level of the config file, or use gitlab-runner –debug run -c \/etc\/gitlab-runner\/config.toml<\/span> to launch a temporary runner.<\/p>\nTo debug Docker API errors, you can use BPF to capture traffic on the UNIX socket, or convert the socket to a TCP listener with socat TCP-LISTEN:12345,reuseaddr,fork UNIX-CLIENT:\/run\/user\/996\/podman\/podman.sock<\/span> , use host = “tcp:\/\/127.0.0.1:12345”<\/span> in the runner config to connect, then use tcpdump -i lo -w docker.pcap tcp port 12345<\/span> to see what happened.<\/p>\nBonus: Use tmpfs to run the containers<\/h1>\n
This will clear all your previous podman data, so make sure you back up important images and containers.<\/p>\n
mkdir -p \/mnt\/gitlab-runner-volatile\r\necho \"tmpfs \/mnt\/gitlab-runner-volatile tmpfs defaults,noatime,size=64g,uid=996,gid=996,mode=0755,huge=always\" >> \/etc\/fstab\r\nmount \/mnt\/gitlab-runner-volatile\r\n\r\nsu - gitlab-runner\r\npodman system reset\r\ncat > ~gitlab-runner\/.config\/containers\/storage.conf <<EOF\r\n[storage]\r\ndriver = \"overlay\"\r\nrunroot = \"\/mnt\/gitlab-runner-volatile\/podman_containers\/runroot\"\r\ngraphroot = \"\/mnt\/gitlab-runner-volatile\/podman_containers\/graphroot\"\r\nrootless_storage_path = \"\/mnt\/gitlab-runner-volatile\/podman_containers\/storage\"\r\n\r\n[storage.options.overlay]\r\nmountopt = \"nodev,metacopy=on\"\r\nmount_program = \"\/usr\/bin\/fuse-overlayfs\"\r\nEOF\r\n\r\nXDG_RUNTIME_DIR=\/run\/user\/$(id -u gitlab-runner) systemctl --user restart podman.service<\/pre>\nAdditional configuration required for SELinux-enabled hosts.<\/a><\/p>\n
\nReferences<\/h1>\n\n- \u4f7f\u7528 CONTAINER-TOOLS API<\/a><\/li>\n
- Why doesn’t my systemd user unit start at boot?<\/a><\/li>\n
- rootless service not work with systemd socket activation<\/a><\/li>\n
- Add support for Podman to GitLab Runner<\/a><\/li>\n
- Configuring container networking with Podman<\/a><\/li>\n
- How to expose a UNIX domain socket directly over TCP<\/a><\/li>\n
- Can’t use tmpfs as runroot for containers<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
GitLab CI runner can be contained in a completely rootless environment. It can start as a non-root user, and work with a rootless Podman instance as a Docker runner. And here is how I achieved it. My CI host configuration: Ubuntu 20.04 Podman 3.4.2 GitLab Runner 14.5.0 A similar procedure can be applied to other […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[],"class_list":["post-304","post","type-post","status-publish","format-standard","hentry","category-linux"],"acf":[],"_links":{"self":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts\/304"}],"collection":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/comments?post=304"}],"version-history":[{"count":19,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts\/304\/revisions"}],"predecessor-version":[{"id":329,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts\/304\/revisions\/329"}],"wp:attachment":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/media?parent=304"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/categories?post=304"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/tags?post=304"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}