diff --git a/gitea/playbook.yml b/gitea/playbook.yml index bde5914..8aed1eb 100755 --- a/gitea/playbook.yml +++ b/gitea/playbook.yml @@ -1,16 +1,57 @@ -################################################# -# DO Community Playbooks: Docker -################################################# --- -- hosts: monoco - become: true +# Gitea as a rootless Podman Quadlet unit on emily, behind the shared Caddy +# unit (caddy/playbook.yml), with git-over-SSH still arriving on the host's own +# port 22 so that no client has to change its remote. +# +# Connects as the container user and runs everything as that user: the Quadlet +# units belong to it, so the connection's PAM session is what makes +# `systemctl --user` work without any environment juggling. +# +# gather_facts is required: podman_user and the home directory come from the +# user_id / user_dir facts. +# +# The rootless-Podman prerequisites (installing podman, enabling lingering, the +# low-port sysctl, creating the Quadlet directory) are deliberately NOT applied +# here. They are the shared task file at caddy/roles/podman/tasks/main.yaml, and +# a host that has already run caddy/playbook.yml has them: +# +# - name: Rootless Podman prerequisites +# ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/podman/tasks/main.yaml" +# +# The variables this play needs are duplicated at the top of vars/default.yml, so +# it stands on its own. (Applying the commented include above would also need +# podman_unprivileged_port_start, which lives in caddy/vars/default.yml.) +# +# Unlike bitwarden/ and tandoor/ there is no postgres role here: this Gitea runs +# on SQLite, which monoco's live app.ini confirms (`DB_TYPE = sqlite3`, +# `PATH = /data/gitea/gitea.db`). The unused Docker-era Postgres variant that +# used to sit beside this file was removed for the same reason. +# +# Run with: ansible-playbook -u tanshu gitea/playbook.yml +- name: Deploy Gitea as a rootless Podman Quadlet + hosts: emily + gather_facts: true vars_files: - vars/default.yml + pre_tasks: + # This must happen before the app unit is started, not after: the generated + # unit references {{ network_name }}.network, and Quadlet fails *unit + # generation* (so `systemctl --user start` reports "unit not found") when a + # referenced .network file does not exist yet. The same include writes the + # Caddy-side drop-in that attaches Caddy to this network. + # + # network_name is a play var, so the include can see it directly. + - name: Register Gitea's network with Caddy + ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml" + roles: - - user - - passthrough - - sshkey - - network + # The host half of git-over-SSH: the `git` login user, the passthrough shim, + # the authorized_keys hook, the sshd drop-in and the one-command sudoers + # rule. Root-only steps elevate individually — see the role for why. + - ssh + # The app: the data directory tree the image's own root phase would leave + # subuid-owned, the Quadlet unit, start + verify. - gitea + # Appends this site's block to the shared Caddyfile and reloads Caddy. - caddy diff --git a/gitea/readme.md b/gitea/readme.md index 2be90cb..4db2fa4 100644 --- a/gitea/readme.md +++ b/gitea/readme.md @@ -1,45 +1,490 @@ -# Gitea on Ubuntu +# Gitea (rootless Podman + Quadlet) -This playbook will install Gitea on an Ubuntu 19.10 machine. -A number of containers will be created with the options specified in the `vars/default.yml` variable file. +Runs Gitea on `emily` as a **rootless Podman** container managed by a **systemd +user unit** (Quadlet), behind the shared [rootless Caddy](../caddy/readme.md), +with git-over-SSH still arriving on the **host's own port 22** so that no client +has to change its remote. Replaces the `docker_container` deployment on `monoco`. + +Two things are worth knowing before anything else: + +- **This Gitea is SQLite.** monoco's live `app.ini` says `DB_TYPE = sqlite3`, + `PATH = /data/gitea/gitea.db`, so there is no `postgres` role here and nothing + to restore into the shared cluster. The unused Docker-era Postgres variant that + used to sit next to the playbook has been deleted. +- **There is no sshd inside the container any more**, and the reason is not a + preference. [Why](#why-there-is-no-sshd-in-the-container) is the heart of this + readme. + +## Ordering + +1. [caddy/playbook.yml](../caddy/playbook.yml) — the reverse proxy, which owns + the Caddyfile this site appends to and is the one play that applies the + rootless-Podman prerequisites. +2. **The data copy below — before the first run of this playbook.** Gitea writes + into `~/podman-data/gitea` the moment it starts, and a first start on an empty + directory serves the *install wizard* instead of this repository's Gitea. +3. This playbook — **run it after the file copy**, not before. + +`postgres/playbook.yml` is not part of this ordering. There is no `postgres` +role in this playbook and no database to attach. + +## Layout + +``` +gitea/ +├── playbook.yml +├── vars/default.yml +└── roles/ + ├── ssh/ # the host half of git-over-SSH: git user, shim, sudoers, sshd drop-in + ├── gitea/ # the data tree the image cannot create, the Quadlet unit, start + verify + └── caddy/ # appends this site's block to the shared Caddyfile +``` + +| On the host | What | +|---|---| +| `~/.config/containers/systemd/gitea.container` | the generated unit (Ansible-templated) | +| `~/.config/containers/systemd/gitea.network` | the app network, written by `caddy/roles/register` | +| `~/.config/containers/systemd/caddy.container.d/50-gitea.conf` | attaches Caddy to that network | +| `~/podman-data/gitea/` | **everything Gitea owns**: `git/` (repositories, LFS), `gitea/` (`app.ini`, `gitea.db`, sessions, avatars, logs), `git/.ssh/authorized_keys` | +| `~/podman-data/gitea/ssh/` | **not Gitea's**: a bare *mount point*, always empty. The in-image sshd's host keys are mounted over it from `gitea-sshd` below, which hides it from the container — so nothing ever writes here | +| `~/podman-data/gitea-sshd/` | scratch space for the *image's* sshd, which this deployment does not use — see below | +| `/home/git/` | the `git` login user's home; **no `.ssh`** — see [The pieces on the host](#the-pieces-on-the-host) | +| `/usr/local/bin/gitea` | the passthrough shim sshd's forced command runs | +| `/usr/local/bin/gitea-container-exec` | `podman exec` wrapper, run as the podman user | +| `/usr/local/bin/gitea-authorized-keys` | what sshd calls to get the git user's keys | +| `/etc/sudoers.d/gitea-ssh-passthrough` | lets `git` run exactly that one wrapper | +| `/etc/ssh/sshd_config.d/60-gitea.conf` | `Match User git` → the keys hook | + +## How git over SSH works now + +### What the Docker deployment did — and why it cannot be repeated + +``` +client ──ssh git@git.tanshu.com:22──► host sshd (monoco) + │ authenticates against /home/git/.ssh/authorized_keys + │ each line is a forced command: + │ command="/usr/local/bin/gitea --config=/data/gitea/conf/app.ini serv key-5",… + ▼ + /usr/local/bin/gitea ← shim, runs as host `git` + │ ssh -p 2222 git@127.0.0.1 "SSH_ORIGINAL_COMMAND=… /usr/local/bin/gitea … serv key-5" + ▼ + sshd inside the container (published 127.0.0.1:2222) + │ authenticated with the host git user's key + │ — the bare, un-forced "Gitea Host Key" line + ▼ + container's gitea: `gitea serv key-5` + └─ key-5 → the user, in Gitea's database → serve the repo +``` + +That works on monoco only because **Docker does not remap uids**: container-uid +1003 *is* host-uid 1003, container-root *is* host-root, so the app, the sshd and +the `git` login user all share one identity and can read each other's files. + +Under rootless Podman with `UserNS=keep-id:uid=1003,gid=1003`, exactly **one** +container uid is shared with the host — 1003 ↔ `tanshu` — and everything else is a +subuid. Verified on `emily` from the running postgres unit, which uses the same +mechanism: + +``` +container 0 → parent 1 → host subuid 165536 +container 70 → parent 0 → host tanshu +``` + +So the container's own sshd has an impossible job: + +| Inside the container | On the host | Consequence | +|---|---|---| +| Gitea (`1003`) | `tanshu` | writes `authorized_keys`, mode `0600` | +| sshd (`0`) | subuid `165536` | **cannot read a `0600` tanshu-owned file** | + +and it cannot be fixed by swapping the mapping: sshd must be root in the +container, while Gitea refuses to be root at all — the image aborts with +`mustNotRunAsRoot() [F] Gitea is not supposed to be run as root`. There is no uid +that satisfies both, and `keep-id` can only map one of them onto the host user. + +### The replacement + +The second hop stops being an ssh and becomes a **`podman exec`**, run as +`tanshu`, the user who owns the container. The public entry point — the host's +sshd on port 22 — is unchanged, and so is every `command=` line Gitea has already +written, because the passthrough shim keeps its path and its arguments: + +``` +client ──ssh git@git.tanshu.com:22──► host sshd (emily) + │ keys from gitea-authorized-keys (AuthorizedKeysCommand, + │ because StrictModes refuses a file git does not own) + ▼ + /usr/local/bin/gitea ← shim, runs as host `git` + │ sudo -n -u tanshu /usr/local/bin/gitea-container-exec \ + │ "$SSH_ORIGINAL_COMMAND" --config=… serv key-5 + ▼ + podman exec -i --user 1003:1003 … gitea \ + /usr/local/bin/gitea --config=/data/gitea/conf/app.ini serv key-5 + └─ env SSH_ORIGINAL_COMMAND="git-upload-pack '…'" + key-5 → the user → serve the repo +``` + +The identity still travels in the command string (`key-5`), so Gitea's per-user +authorisation is exactly what it was. What is gone: the 2222 publish, the +container's sshd as a hop, `/data/ssh` in the data tree, and the "Gitea Host Key" +that only existed to authenticate that hop. + +### Why there is no sshd in the container + +- sshd must run as the container's root, which under `keep-id` is a subuid. +- Gitea refuses to run as root, so it runs as container-uid 1003 = `tanshu`. +- Gitea maintains `authorized_keys` itself, as `tanshu`, mode `0600`. +- A subuid cannot read it. `StrictModes` would also reject it. +- Rewriting the modes is not a fix: `sshd` ignores world-readable host keys, and + the file is rewritten by Gitea after every key change. +- `podman exec` needs no uid to be shared with the container at all, so it works. + +The image still *starts* an sshd of its own (`/etc/s6/openssh`), because that is +baked into the image. It is unpublished and nothing connects to it, but it does +want host keys, and it cannot write them into the data tree for the same subuid +reason. It therefore gets `~/podman-data/gitea-sshd`, a scratch directory mounted +over `/data/ssh` **outside** the data tree — so no subuid-owned file ever lands in +`~/podman-data/gitea`, a plain `tar` of the data tree stays complete, and the +play's "nothing in here belongs to another user" check stays meaningful. + +> If you prefer no stray directory at all, a tmpfs works in principle: +> `Mount=type=tmpfs,destination=/data/ssh,tmpfs-mode=1777` instead of the +> `Volume=` line. **Not verified on emily** — try it in a scratch copy of the unit +> and watch `podman logs gitea` for the image's openssh service coming up. + +## The pieces on the host + +**The `git` login user** exists only so sshd has someone to authenticate as. It +has `/bin/bash` because sshd runs a forced command through the login shell +(`/bin/bash -c '…'`); a nologin shell would break git-over-SSH outright. Its uid +is deliberately *not* pinned: on monoco it was 1003 because it had to equal the +container's git uid, and under `keep-id` that pairing no longer exists. + +**There is no `/home/git/.ssh`.** It used to hold the shim's client key (the +"Gitea Host Key", now obsolete — nothing authenticates over ssh any more) and was +bind-mounted into the container as `/data/git/.ssh`. The file that matters now +lives in the data tree, where Gitea writes it. The migration below moves it there. + +**`/usr/local/bin/gitea`** is the shim. Its path is a contract: it is baked into +every `command=` line Gitea has already written, so renaming it locks every user +out until Gitea rewrites the file. + +**`/usr/local/bin/gitea-container-exec`** is the `podman exec`, with the uid, +`HOME`, `GITEA_CUSTOM`, `SSH_ORIGINAL_COMMAND` and `XDG_RUNTIME_DIR` that the +container's sshd used to provide. `XDG_RUNTIME_DIR` is set explicitly because +`sudo -u` creates no PAM session — the same trick +[caddy/readme.md](../caddy/readme.md) shows for `sudo -u … systemctl --user`. + +**`/etc/sudoers.d/gitea-ssh-passthrough`** grants `git` exactly one command as +`tanshu`. It is installed with `visudo -cf` validation, because a syntax error in +`/etc/sudoers.d` does not break one rule, it breaks *every* `sudo` on the host. +The trust boundary is worth stating plainly: whoever can obtain a `git` shell on +this host can run containers as `tanshu`. Nothing gets that shell — every key +Gitea accepts carries a forced command — and the hook below is what keeps it that +way. + +**`/etc/ssh/sshd_config.d/60-gitea.conf`** points sshd at +`gitea-authorized-keys` for `Match User git`. It has to be a hook rather than the +plain file because the file belongs to `tanshu`, which is exactly what +`StrictModes` refuses — and `StrictModes` cannot be relaxed for one user: +OpenSSH rejects it inside a `Match` block (`Directive 'StrictModes' is not +allowed within a Match block`, confirmed on emily, OpenSSH 10.2p1). +`AuthorizedKeysCommand` output is used without an ownership check, so that is the +mechanism. The drop-in is validated with a full `sshd -t` before sshd is +reloaded, and removed again if it does not parse. + +**`/usr/local/bin/gitea-authorized-keys`** prints Gitea's file, filtered to the +lines that force the shim. The unforced "Gitea Host Key" line is deliberately +*dropped*: nothing needs it now, and printing it would hand an interactive `git` +login — and with it rootless Podman, via the sudoers rule — to anyone holding a +copy of that key, a backup included. + +## Who runs as whom + +`UserNS=keep-id:uid=1003,gid=1003` plus `Environment=USER_UID/USER_GID=1003`, all +three from `gitea_uid` in [vars/default.yml](vars/default.yml). Gitea therefore +writes as container-1003, which is `tanshu` on the host: everything under +`~/podman-data/gitea` is `tanshu`-owned, and plain `tar`/`rsync` backups keep +working. + +**`User=0` is the fourth line, and without it nothing starts.** `keep-id` does not +only map the uid, it also runs the container *process* as it — so the whole +container came up as `git`, while the image's start-up is written to begin as +root: `/usr/bin/entrypoint` rewrites `/etc/passwd`, `/etc/s6/gitea/setup` (sourced +by `/etc/s6/gitea/run` before it `su-exec git`s the server) prepares `/data`, and +the image's own CMD is `s6-svscan /etc/s6`, which has to create +`.s6-svscan/lock` inside root-owned `/etc/s6`. Denied all three, the container +exited immediately and `Restart=always` turned that into a crash loop — which +`is-active` reports as `activating`, not as a failure. `User=0` restores root for +the start-up and changes nothing about the mapping above: the *server* still runs +as `git` through `su-exec`, which is what satisfies Gitea's own +`mustNotRunAsRoot` check. (`nextcloud.container` carries the same pair for the +same reason.) + +The consequence to internalise: **container root is a subuid on the host.** It +holds the usual capabilities *inside* its user namespace, so the start-up's +`mkdir`, `chmod` and `chown` all succeed — and what they create is left owned by +that subuid. The image chowns the directories it knows about back to `git` +(`chown -R ${USER}:git /data/gitea` and `/data/git` in `/etc/s6/gitea/setup`), but +not everything, so [roles/gitea](roles/gitea/tasks/main.yaml) creates the tree up +front — `gitea/`, `gitea/conf/`, `gitea/log/`, `git/`, `git/.ssh/`, `ssh/` — owned +by `tanshu`, with the modes those scripts would have set. Two of them are not +housekeeping at all: + +- **`git/.ssh/environment`** is written by the play rather than left to the setup + script, because the setup's copy is the one file under `/data` it never chowns + back to `git`. +- **`ssh/`** is the mount point for the in-image sshd's host keys. It is hidden + inside the container by that mount, so nothing ever chowns it: if Podman has to + create it, it lands in the data tree owned by the subuid, and the check at the + end of the role fails on it. That is exactly what happened before this was added. + +This is the same trap [postgres/readme.md](../postgres/readme.md) documents for +`PGDATA`, arrived at from the other direction. + +### When it does not start + +The container's start-up output is in the journal, and the play prints it for you: +every step from `Start Gitea` through the health check is in one block whose +`rescue` dumps `systemctl status`, `journalctl` and `podman logs`. The lines that +have actually shown up, and what each means: + +| In the journal | Cause | +|---|---| +| `s6-svscan: fatal: unable to open .s6-svscan/lock: Permission denied` | the container process is not root — `User=0` is missing | +| `sed: can't create temp file '/etc/passwdXXXXXX': Permission denied` | same cause: the entrypoint may not rewrite `/etc/passwd` | +| `DB not initialized` / the install page in a browser | the copy from monoco has not run, so this is an empty Gitea | + +Beyond those, `podman logs gitea` on a healthy container is **clean** — no +permission errors at all. An earlier draft of this readme listed four +`chmod … Operation not permitted` lines as expected noise; they do not occur, and +if you see something like them they are worth reading rather than skimming. ## Settings -- `create_containers`: number of containers to create. -- `default_container_name`: default name for new containers. -- `default_container_image`: default image for new containers. -- `default_container_command`: default command to run on new containers. +| Variable | Purpose | +|---|---| +| `podman_user`, `podman_data_dir`, `quadlet_dir` | Rootless-Podman contract, duplicated from [caddy/vars/default.yml](../caddy/vars/default.yml) | +| `container_name` | Container, unit and network prefix — `gitea` | +| `image` | `docker.io/gitea/gitea:latest`, fully qualified on purpose | +| `host_directory` | Where all state lives (`~/podman-data/gitea`) | +| `container_port` | `3000`, the port inside the container Caddy proxies to | +| `gitea_uid` / `gitea_gid` | `1003` — the image's own `git` account. Drives `UserNS`, `USER_UID`/`USER_GID` and the wrapper's `--user`; see [Who runs as whom](#who-runs-as-whom) | +| `gitea_custom` | `/data/gitea`, the image's `GITEA_CUSTOM` | +| `timezone` | `TZ` for log timestamps only; Gitea stores each user's own timezone | +| `health_cmd` | The unit's `HealthCmd` — `/api/healthz`, Gitea's unauthenticated probe | +| `container_sshd_hostkeys` | Scratch dir for the image's unused sshd, mounted over `/data/ssh` | +| `git_user`, `git_home` | The SSH login account (`git`, `/home/git`) | +| `shim_path` | `/usr/local/bin/gitea` — a **contract**, not a knob | +| `podman_exec_wrapper`, `authorized_keys_command` | The two helper scripts | +| `sshd_dropin_path`, `sshd_service`, `sudoers_path` | `/etc/ssh/sshd_config.d/60-gitea.conf`, `ssh`, `/etc/sudoers.d/gitea-ssh-passthrough` | +| `host`, `caddy_name`, `caddyfile_path`, `snippet_block` | The Caddy side, as in every service playbook | +Keep the three `podman_*`/`quadlet_dir` vars in sync with +[caddy/vars/default.yml](../caddy/vars/default.yml) — they are duplicated rather +than imported because a play cannot include Caddy's vars file wholesale (it also +sets `host_directory`, which every service defines for itself). -## Running this Playbook - -Quick Steps: - -### 1. Obtain the playbook -```shell -git clone https://github.com/do-community/ansible-playbooks.git -cd ansible-playbooks/docker_ubuntu1804 -``` - -### 2. Customize Options - -```shell -nano vars/default.yml -``` - -```yml -#vars/default.yml ---- -create_containers: 4 -default_container_name: docker -default_container_image: ubuntu -default_container_command: sleep 1d -``` - -### 3. Run the Playbook +## Running ```command -ansible-playbook -l [target] -i [inventory file] -u [remote user] playbook.yml +ansible-playbook -u tanshu gitea/playbook.yml ``` -For more information on how to run this Ansible setup, please check this guide: [How to Use Ansible to Install and Set Up Docker on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-use-ansible-to-install-and-set-up-docker-on-ubuntu-18-04). +The play targets `emily` directly, so the inventory must contain it — the one in +`prepare-server/` does, and its `ansible.cfg` names it. + +This play applies **only** this app's units, data directories and host-side SSH +plumbing. The rootless-Podman prerequisites — installing podman, +`loginctl enable-linger`, the low-port sysctl and the Quadlet directory — are the +shared task file at +[caddy/roles/podman/tasks/main.yaml](../caddy/roles/podman/tasks/main.yaml), and +are **not** run here. A host that has already run `caddy/playbook.yml` has them. + +`systemctl --user enable gitea` is neither needed nor possible: the unit is +*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself. + +## Migrating the data from monoco + +Everything here is about **files Gitea owns**. There is no database dump to take: +this Gitea is SQLite, and `gitea.db` travels inside the tree. Run the commands +from the machine that holds the `monoco` and `emily` SSH aliases. + +Two things in the source tree are deliberately **not** copied: + +- **`/var/lib/gitea/ssh`** — host keys for the sshd inside the container. That + sshd is no longer a hop, nothing connects to it, and the new deployment lets it + generate fresh keys in `~/podman-data/gitea-sshd`. The Docker-era audit told you + to copy them so clients would not see host-key warnings; with the container's + sshd out of the path, no client ever sees them. +- **`/home/git/.ssh/id_ed25519` and friends** — the "Gitea Host Key" that + authenticated the old second hop. Nothing authenticates over ssh now. Only + `authorized_keys` (and optionally `environment`) is carried across, into the + container's data tree, which is where Gitea expects to maintain it. + +```bash +# 1. Look before you copy: what is there, and how big is it. `ls -ln` shows +# numeric owners, because the old container wrote as uid 1003 (`git`). +ssh monoco 'du -sh /var/lib/gitea/*; ls -ln /var/lib/gitea' +ssh monoco 'sudo ls -ln /home/git/.ssh' + +# 2. Stop the old container so nothing writes while you copy. SQLite in WAL mode +# is exactly the thing the backup plan warns about — a live copy can be torn. +# Leave it stopped: emily is about to become the only writer, and two live +# copies behind the same hostname would diverge. +ssh monoco 'sudo docker stop gitea' + +# 3. Create the destination tree (the playbook would create it too, but the copy +# needs somewhere to land). +ssh emily 'mkdir -p ~/podman-data/gitea/git/.ssh' + +# If the playbook has already run here — it says so when it does — the app has +# left a fresh `gitea/conf/app.ini` (and possibly logs) behind, and the digests +# in step 6 will not match. Clear it first: +# ssh emily 'systemctl --user stop gitea.service; find ~/podman-data/gitea/gitea ~/podman-data/gitea/git -mindepth 1 -delete' + +# 4. Copy the data tree, minus the obsolete container-sshd host keys. +# tar streams it across the workstation; rsync cannot copy remote-to-remote in +# one command. +# +# `set -o pipefail` matters: without it the pipeline reports only the +# *receiving* tar's status, so a read error or a dropped stream on the source +# side exits 0 and leaves a silently short copy. +# +# The source needs sudo — the files belong to `git`, not to you. +set -o pipefail +ssh monoco 'sudo tar --exclude="./ssh" -C /var/lib/gitea -cf - .' \ + | ssh emily 'tar --no-same-owner -C ~/podman-data/gitea -xf -' + +# 5. Copy the authorized_keys Gitea maintains. It lives outside /var/lib/gitea on +# the old host, because it was bind-mounted in from the git user's home. +set -o pipefail +ssh monoco 'sudo tar -C /home/git/.ssh -cf - authorized_keys environment' \ + | ssh emily 'tar --no-same-owner -C ~/podman-data/gitea/git/.ssh -xf -' + +# 6. Verify by content, over both trees, with the same exclusions on each side so +# the digests are comparable. `.ssh` is excluded here because it is compared +# on its own in the next step, and `ssh/` never left monoco. +ssh monoco 'cd /var/lib/gitea && sudo find ./git ./gitea ! -path "./git/.ssh*" -type f -print0 \ + | sort -z | xargs -0 -r sudo sha256sum | sha256sum' +ssh emily 'cd ~/podman-data/gitea && find ./git ./gitea ! -path "./git/.ssh*" -type f -print0 \ + | sort -z | xargs -0 -r sha256sum | sha256sum' +# identical digests = a byte-faithful copy + +ssh monoco 'sudo sha256sum /home/git/.ssh/authorized_keys | cut -d" " -f1' +ssh emily 'sha256sum ~/podman-data/gitea/git/.ssh/authorized_keys | cut -d" " -f1' + +# 7. Nothing may end up owned by root or by a subuid, and sshd's hook must be +# able to read the file it is about to be handed. +ssh emily 'find ~/podman-data/gitea ! -user tanshu' # empty = correct +ssh emily 'ls -ln ~/podman-data/gitea/git/.ssh/authorized_keys' + +# 8. Deploy — after the copy, so Gitea finds the real data on first start. +cd prepare-server && ansible-playbook -u tanshu ../gitea/playbook.yml +``` + +If the link is slow or drops, do it in two hops instead: resumable, and it shows +progress. + +```bash +rsync -aP --exclude='ssh' monoco:/var/lib/gitea/ ~/gitea-copy/ +rsync -aP ~/gitea-copy/ emily:~/podman-data/gitea/ +rsync -aP monoco:'/home/git/.ssh/{authorized_keys,environment}' ~/gitea-ssh-copy/ +rsync -aP ~/gitea-ssh-copy/ emily:~/podman-data/gitea/git/.ssh/ +``` + +Rolling back means rolling back all of it: `ssh monoco 'sudo docker start gitea'` +is only meaningful while monoco's data is still intact, and it would then be +writing to a `gitea.db` that emily's copy has already moved past. **Never run +both at once.** + +## Verification + +```bash +systemctl --user status gitea +journalctl --user -u gitea -f +podman logs gitea # clean — see "When it does not start" above + +# What the playbook checks for you: +podman healthcheck run gitea # /api/healthz +find ~/podman-data/gitea ! -user tanshu # empty + +# The unit is where it should be, and the image's sshd has its scratch space: +podman inspect gitea --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}' + +# The host's SSH half, piece by piece: +sudo sshd -T -C user=git | grep -i authorizedkeys # the hook, and only for git +sudo -u tanshu /usr/local/bin/gitea-authorized-keys | head -3 +sudo -u tanshu /usr/local/bin/gitea-authorized-keys | grep -c '^ssh-' # 0 — the bare host-key line is dropped +sudo visudo -cf /etc/sudoers.d/gitea-ssh-passthrough +# the wrapper reaches the container as the right user: +sudo -u tanshu /usr/local/bin/gitea-container-exec "" /usr/local/bin/gitea --version +# and the git user is allowed to run exactly that wrapper: +sudo -u git sudo -n -u tanshu /usr/local/bin/gitea-container-exec "" /usr/local/bin/gitea --version + +# The site, end to end. +curl -sSI https://git.tanshu.com +``` + +Then, from a client whose key is registered in Gitea: + +```bash +ssh -T git@git.tanshu.com # "Hi ! You've successfully authenticated…" +git clone git@git.tanshu.com:tanshu/playbooks.git +``` + +and push to a repository you own — that is the check that per-user authorisation +still works, because the identity travels in the `serv key-N` argument and *not* +in the shim's own credentials. Then exercise the web UI and confirm a repository +list renders, which is the check that `gitea.db` came across. + +Finally, reboot the host and confirm the unit returns unattended — without +`loginctl enable-linger tanshu` it will not, and the failure is silent. + +## Notes and gotchas + +- **`DISABLE_SSH` must stay `false`, and Gitea's built-in SSH server must stay + off** (`START_SSH_SERVER` absent or `false`) in `app.ini`. Gitea only maintains + `authorized_keys` for an *external* sshd — which the host's sshd is here. Turn + the built-in server on and Gitea stops writing the file the host's sshd reads, + and git-over-SSH stops working with no error anywhere obvious. The copied + `app.ini` already has this right. +- **`SSH_PORT = 22` stays as it is.** It is what makes Gitea offer + `git@git.tanshu.com:user/repo.git` rather than a URL with a port in it, and the + host's sshd is what answers on 22. `SSH_LISTEN_PORT = 22` refers to the + container's own sshd port, which this deployment does not use. +- **The `git` user can reach rootless Podman as `tanshu`** through one sudoers + rule. That is the price of a two-user design and it is bounded to one root-owned + wrapper; see [The pieces on the host](#the-pieces-on-the-host). +- **The authorized_keys filter is a format dependency.** If Gitea ever stops + writing `command="/usr/local/bin/gitea …` lines, the hook prints nothing and + git-over-SSH breaks while HTTPS keeps serving. Diagnose with + `sudo -u tanshu /usr/local/bin/gitea-authorized-keys` and + `podman exec gitea cat /data/git/.ssh/authorized_keys`. +- **Pushing over SSH runs no hooks through the host user.** Hooks execute inside + the container, as `git` = `tanshu`, so a repository hook can write anywhere + `tanshu` can. Same as before the move, but worth knowing before enabling + server-side hooks. +- **`latest` floats.** Neither the old playbook nor this one pins a digest, so an + image bump is a normal `podman pull` away and may run new database migrations on + first start. +- **The old backup script's `docker exec -u git gitea gitea dump …` needs + rewriting** for this deployment: `sudo podman exec` from root's crontab cannot + enter another user's rootless containers, so anything scheduled belongs in a + **systemd user timer** next to the Quadlet units. See the Backups section of + [postgres/readme.md](../postgres/readme.md) for the shape of it. +- **DNS.** `git.tanshu.com` must resolve to `emily` before Caddy can obtain a + certificate for it. +- **Do not restart the container on monoco** while emily is serving. It would + write to the old data tree, which is being kept intact purely as a rollback. + +## Not done here + +- Rewriting the backup script for rootless Podman and moving it to a systemd user + timer (above). +- Rotating the secrets that live in `app.ini` in plaintext — `SECRET_KEY`, + `INTERNAL_TOKEN`, `LFS_JWT_SECRET`, `JWT_SECRET` — and the fleet-wide + `123456`-style passwords elsewhere in this repo. +- Templating `app.ini`. Gitea's own configuration travels with the data, and it + carries those secrets; a fresh instance with no data gets the image's install + wizard instead, which is a reasonable outcome but not a supported path here. diff --git a/gitea/roles/caddy/handlers/main.yaml b/gitea/roles/caddy/handlers/main.yaml index bc77d4f..d12aa38 100644 --- a/gitea/roles/caddy/handlers/main.yaml +++ b/gitea/roles/caddy/handlers/main.yaml @@ -1,5 +1,31 @@ --- -# Handlers for Caddy role +# Caddy is its own playbook (caddy/playbook.yml) and a rootless Quadlet unit +# there. These handlers run as the connection user, so `systemctl --user` and +# `podman exec` work without any XDG_RUNTIME_DIR juggling. + +# Notified by this role when the Caddyfile changes. A reload keeps existing +# connections; a restart would briefly drop every hosted site. +# +# Note the config path: /etc/caddy/Caddyfile is the mount inside the container, +# not {{ caddyfile_path }}, which is the host side of it. - name: Reload Caddy configuration - ansible.builtin.command: "docker exec -w /etc/caddy {{ caddy_container }} caddy reload" + ansible.builtin.command: >- + podman exec {{ caddy_name }} caddy reload --config /etc/caddy/Caddyfile + changed_when: true listen: "Reload Caddy" + +# Notified by caddy/roles/register when this service's network unit, or the +# drop-in that attaches Caddy to it, is written. Quadlet has no live equivalent +# of `docker network connect`, so the container has to be recreated — a brief +# restart of every hosted site, not just this one. See caddy/readme.md, +# "Trade-off worth knowing". +# +# Defined after the reload above so that when both are notified the cheap config +# reload happens first. +- name: Restart Caddy + ansible.builtin.systemd_service: + name: "{{ caddy_name }}.service" + scope: user + state: restarted + daemon_reload: true + listen: Restart Caddy diff --git a/gitea/roles/caddy/tasks/main.yaml b/gitea/roles/caddy/tasks/main.yaml index 6c99a6a..ffbe0d8 100644 --- a/gitea/roles/caddy/tasks/main.yaml +++ b/gitea/roles/caddy/tasks/main.yaml @@ -1,22 +1,22 @@ --- -- name: Read current Caddyfile - ansible.builtin.slurp: - path: "{{ caddyfile_path }}" - register: caddyfile_raw +# Append this site's block to the shared Caddyfile. +# +# The Caddyfile is a bind mount into the Caddy container, so Caddy sees the +# change as soon as the file is written; the handler only asks Caddy to re-read +# it. That is a reload, not a restart, so the other hosted sites are untouched. +# +# blockinfile is idempotent on its own via the marker, so there is no need to +# slurp the file and compare first — which also matters because slurp *fails* +# when the Caddyfile does not exist yet, on a host where the Caddy playbook has +# not run. -- name: Decode Caddyfile content - set_fact: - caddyfile_content: "{{ caddyfile_raw['content'] | b64decode }}" - -- name: Check if snippet already exists - set_fact: - snippet_present: "{{ snippet_block in caddyfile_content }}" - -- name: Add snippet if missing +- name: "Add the {{ host }} site block to the Caddyfile" ansible.builtin.blockinfile: path: "{{ caddyfile_path }}" - marker: "# {mark} Ansible managed Caddy snippet for {{ http_host }}" + marker: "# {mark} Ansible managed Caddy snippet for {{ host }}" block: "{{ snippet_block }}" create: yes - when: not snippet_present - notify: "Reload Caddy" + owner: "{{ podman_user }}" + group: "{{ podman_user }}" + mode: '0644' + notify: Reload Caddy diff --git a/gitea/roles/gitea/handlers/main.yaml b/gitea/roles/gitea/handlers/main.yaml new file mode 100644 index 0000000..ffc633f --- /dev/null +++ b/gitea/roles/gitea/handlers/main.yaml @@ -0,0 +1,19 @@ +--- +# All handlers run as the connection user ({{ podman_user }}), so systemctl +# --user works without setting XDG_RUNTIME_DIR. + +- name: Reload systemd user units + ansible.builtin.command: systemctl --user daemon-reload + changed_when: true + listen: Reload systemd user units + +# A changed unit file changes the generated unit, so Gitea is recreated. All of +# its state is on the bind mount or in the SQLite database, so a recreate loses +# nothing — but it is a few seconds of downtime for the one site. +- name: Restart Gitea + ansible.builtin.systemd_service: + name: "{{ unit_name }}" + scope: user + state: restarted + daemon_reload: true + listen: Restart Gitea diff --git a/gitea/roles/gitea/tasks/main.yaml b/gitea/roles/gitea/tasks/main.yaml index e89700d..a8b6697 100644 --- a/gitea/roles/gitea/tasks/main.yaml +++ b/gitea/roles/gitea/tasks/main.yaml @@ -1,36 +1,211 @@ --- -# Tasks for Gitea role -- name: Pull Gitea image - docker_image: - name: "{{ container_image }}" - source: pull - state: present - force_source: yes +# Gitea as a rootless Podman Quadlet unit. +# +# Everything here runs as {{ podman_user }} — the connection user. No become, no +# XDG_RUNTIME_DIR: the SSH connection gave us a real PAM session, so systemd's +# user manager is reachable through the standard environment. -- getent: - database: passwd - key: "{{ user }}" - split: ":" +# The data tree is created here rather than left to the image, and that is not +# housekeeping. With keep-id:uid={{ gitea_uid }} the container process runs as +# `git`, so /usr/bin/entrypoint is the *only* part of the image's start-up that +# root can do — and it needs root for /etc/passwd, /etc/s6 and this tree. +# +# Container root is a subuid on the host, but inside its user namespace it holds +# the usual capabilities, so `mkdir`, `chmod` and `chown` all succeed; what it +# creates is simply left owned by that subuid. The image chowns the directories it +# knows about back to `git` — `chown -R ${USER}:git /data/gitea` and `/data/git` +# in /etc/s6/gitea/setup, which is where app.ini comes from — so those work either +# way. The things it does *not* chown are what this task is for, and they are the +# ones that would otherwise be subuid-owned and invisible to the app and to a +# plain tar backup: +# +# gitea/ app.ini, gitea.db, sessions, avatars, indexers +# gitea/conf where the entrypoint would generate app.ini +# gitea/log Gitea's file log +# git/ repositories, LFS, and the git account's HOME +# git/.ssh authorized_keys — written by Gitea, read by the host's sshd +# ssh/ only ever a *mount point* — see below +# +# The sshd *inside* the image is the exception; it cannot use this tree at all. +# See {{ container_sshd_hostkeys }} below. +- name: Ensure the Gitea data directory exists + ansible.builtin.file: + path: "{{ host_directory }}" + state: directory + owner: "{{ podman_user }}" + group: "{{ podman_user }}" + mode: '0755' -- name: Create gitea container - docker_container: - name: "{{ container_name }}" - image: "{{ container_image }}" - state: started - restart_policy: "unless-stopped" - cpus: .10 - memory: "512M" - memory_swap: "512M" - env: - USER_UID: "{{ ansible_facts['getent_passwd'][user][1] }}" - USER_GID: "{{ ansible_facts['getent_passwd'][user][2] }}" - published_ports: - - "127.0.0.1:3000:3000" - - "127.0.0.1:2222:22" - volumes: - - /var/lib/gitea:/data - - /etc/timezone:/etc/timezone:ro - - /etc/localtime:/etc/localtime:ro - - "/home/{{ user }}/.ssh/:/data/git/.ssh" - networks: - - name: "{{ docker_network }}" +- name: Ensure the directories the image expects under /data exist + ansible.builtin.file: + path: "{{ host_directory }}/{{ item.path }}" + state: directory + owner: "{{ podman_user }}" + group: "{{ podman_user }}" + mode: "{{ item.mode }}" + loop: + - { path: gitea, mode: '0755' } + - { path: gitea/conf, mode: '0755' } + - { path: gitea/log, mode: '0755' } + - { path: git, mode: '0755' } + # 0700 because that is what the image's own setup script chmods it to + # (`chmod 700 /data/git/.ssh` in /etc/s6/gitea/setup, which runs as root + # before `su-exec git`). Set here so the mode never depends on that script + # having run. + - { path: git/.ssh, mode: '0700' } + # The in-image sshd's host keys are mounted over /data/ssh from + # {{ container_sshd_hostkeys }}, so this directory is never written to — it + # has to *exist*, because it is the mount point. If it does not, Podman + # creates it during container set-up, inside the container's user namespace, + # and it lands here owned by the subuid rather than by {{ podman_user }}: + # + # drwxr-xr-t 2 165536 165536 4096 ssh + # + # which the check at the end of this role then (correctly) fails on. Creating + # it first is what keeps the data tree wholly {{ podman_user }}'s. + - { path: ssh, mode: '0755' } + +# The image's setup creates this file if it is missing, as root — and it is the +# one file under /data the setup never chowns back to `git`, so on a fresh host it +# would be left subuid-owned. Writing it here, as {{ podman_user }} and 0600, +# both avoids that and states the value the image's own environment carries. +- name: Write the container-side .ssh/environment file + ansible.builtin.copy: + dest: "{{ host_directory }}/git/.ssh/environment" + content: | + GITEA_CUSTOM={{ gitea_custom }} + owner: "{{ podman_user }}" + group: "{{ podman_user }}" + mode: '0600' + +# The image always starts an sshd of its own inside the container, and that sshd +# wants host keys. This deployment does not use it — git-over-SSH arrives on the +# host's sshd on port 22 — but it has to have somewhere to write, and the process +# that runs it maps to a subuid that cannot touch the data directory. Keeping it +# out here means no subuid-owned file ever lands in {{ host_directory }}, so a +# plain tar/rsync of the data tree stays complete and error-free, and the check +# at the end of this role stays meaningful. +# +# 0777 is deliberate and bounded: /home/{{ podman_user }} is 0750, so no other +# host user can reach this path at all. The mode only matters inside the userns. +- name: Ensure the scratch host key directory exists + ansible.builtin.file: + path: "{{ container_sshd_hostkeys }}" + state: directory + owner: "{{ podman_user }}" + group: "{{ podman_user }}" + mode: '0777' + +- name: Install the Gitea Quadlet unit + ansible.builtin.template: + src: gitea.container.j2 + dest: "{{ quadlet_dir }}/{{ container_name }}.container" + owner: "{{ podman_user }}" + group: "{{ podman_user }}" + mode: '0644' + notify: + - Reload systemd user units + - Restart Gitea + +# Quadlet honours [Install] WantedBy= when it generates the unit, so +# `systemctl --user enable` is neither needed nor possible on a generated unit. +# +# All three steps are in one block because they fail the same way and want the +# same evidence. `systemctl start` reports success as soon as it has handed the +# unit to systemd, so a container that dies a second later — a `User=0` missing, +# an unwritable /data, a subuid that cannot run the image's start-up — surfaces +# only as `activating` from `is-active`, or as a health check that never passes, +# with the reason in the journal and nowhere else. Restart=always makes that a +# crash loop rather than an error. +- name: Start and verify Gitea + block: + - name: Start Gitea + ansible.builtin.systemd_service: + name: "{{ unit_name }}" + scope: user + state: started + daemon_reload: true + + # Retried rather than checked once: podman reports `activating` for the + # interval between systemd starting the container and its readiness + # notification. + - name: Verify Gitea is running + ansible.builtin.command: "systemctl --user is-active {{ unit_name }}" + register: gitea_active + until: gitea_active.stdout == 'active' + retries: 12 + delay: 5 + changed_when: false + + # `is-active` says the container is running; it does not say the app can + # serve. The image ships no HEALTHCHECK, so the unit defines one (HealthCmd=, + # see the template) and this runs exactly that command once, synchronously. + - name: Wait for Gitea to pass its health check + ansible.builtin.command: "podman healthcheck run {{ container_name }}" + register: gitea_health + until: gitea_health.rc == 0 + retries: 30 + delay: 5 + changed_when: false + rescue: + # systemd reports "control process exited with error code" and nothing else, + # which is useless for the usual failures. The journal is where the image's + # own start-up output is — that is where a s6 or entrypoint permission + # failure shows up. + - name: Collect Gitea diagnostics + ansible.builtin.command: "{{ item }}" + loop: + - "systemctl --user status {{ unit_name }} --no-pager" + - "journalctl --user -u {{ unit_name }} --no-pager -n 40" + - "podman logs --tail 40 {{ container_name }}" + register: gitea_diag + changed_when: false + failed_when: false + + - name: Show Gitea diagnostics + ansible.builtin.debug: + msg: "{{ gitea_diag.results | map(attribute='stdout') | list }}" + + - name: Fail + ansible.builtin.fail: + msg: "Gitea did not start — see the diagnostics above." + +# A wrong keep-id is the one failure here that does not look like one: Gitea +# still serves, but from files it owns as an unmapped subuid on the host — files +# that then disappear from a tar backup and cannot be deleted without +# `podman unshare`. Check the whole tree, not a sample. +# +# Deliberately not `failed_when: false`: if `find` cannot read something in the +# data directory, the tree is not provably clean, and that is the finding. +- name: List anything in the data tree not owned by {{ podman_user }} + ansible.builtin.command: >- + find {{ host_directory }} ! -user {{ podman_user }} + register: gitea_stray_owners + changed_when: false + +- name: Fail if the data tree holds files owned by another user + ansible.builtin.assert: + that: gitea_stray_owners.stdout_lines | length == 0 + fail_msg: >- + {{ gitea_stray_owners.stdout_lines | join(', ') }} — every file under + {{ host_directory }} must belong to {{ podman_user }}. A subuid in there + means the uid Gitea writes as and the uid mapped onto {{ podman_user }} + have drifted apart: compare gitea_uid in vars/default.yml with UserNS and + USER_UID/USER_GID in {{ quadlet_dir }}/{{ container_name }}.container. + +- name: Check whether the data has been restored + ansible.builtin.stat: + path: "{{ host_directory }}/gitea/gitea.db" + register: gitea_db + +# Not a failure: a fresh instance legitimately has no database yet. But on a +# migration it means the copy in readme.md has not run, and the instance that is +# now serving is an empty installer rather than this repository's Gitea. +- name: Report a data directory with no database in it + ansible.builtin.debug: + msg: >- + {{ host_directory }}/gitea/gitea.db is missing, so this Gitea has no data: + it will serve its install page and write a fresh app.ini. Expected only on + a brand-new instance — a migrated one is given its data first. See + readme.md, "Migrating the data from monoco". + when: not gitea_db.stat.exists diff --git a/gitea/roles/gitea/templates/gitea.container.j2 b/gitea/roles/gitea/templates/gitea.container.j2 new file mode 100644 index 0000000..4f1a24d --- /dev/null +++ b/gitea/roles/gitea/templates/gitea.container.j2 @@ -0,0 +1,83 @@ +# Gitea — rootless Podman Quadlet unit. +# Managed by Ansible (gitea/roles/gitea). Do not edit by hand. +# +# Joins {{ network_name }}.network, written by caddy/roles/register — the same +# include that attaches Caddy to it. Caddy reaches Gitea by container name, so +# nothing is published on the host for HTTP. + +[Unit] +Description=Gitea +Documentation=https://docs.gitea.com/ +After=network-online.target +Wants=network-online.target + +[Container] +ContainerName={{ container_name }} +Image={{ image }} + +# Gitea refuses to run as root (`mustNotRunAsRoot`), so the app cannot simply be +# container-root the way Caddy and Audiobookshelf are. Instead container uid +# {{ gitea_uid }} — the image's own `git` account — is mapped onto +# {{ podman_user }}, which keeps every file Gitea writes owned by +# {{ podman_user }} on the host: plain tar/rsync backups, no subuids, no +# `podman unshare`. The cost is that the image's *root* phases run as a subuid, +# which is why roles/gitea creates the data tree up front. +# +# USER_UID/USER_GID are not decoration: they make the entrypoint keep the +# account at {{ gitea_uid }} instead of the image default, which is what holds +# the file layout and this mapping together. All of it comes from gitea_uid in +# vars/default.yml — change one and the others have to follow. +UserNS=keep-id:uid={{ gitea_uid }},gid={{ gitea_gid }} +Environment=USER_UID={{ gitea_uid }} +Environment=USER_GID={{ gitea_gid }} + +# keep-id also runs the container *process* as that mapped uid, i.e. as `git`, +# while the image's start-up is written to begin as root: `s6-svscan`, which is +# the image's own CMD, creates `.s6-svscan/lock` inside root-owned /etc/s6, and +# /usr/bin/entrypoint rewrites /etc/passwd and /etc/group to apply +# USER_UID/USER_GID. Denied both, the container died with +# +# sed: can't create temp file '/etc/passwdXXXXXX': Permission denied +# s6-svscan: fatal: unable to open .s6-svscan/lock: Permission denied +# +# and Restart=always turned that into a crash loop — which `is-active` reports as +# `activating`, not as a failure. User=0 forces the process back to root, and the +# mapping above is untouched by it: the server itself still runs as `git`, which +# is what satisfies mustNotRunAsRoot. (nextcloud.container carries the same pair +# for the same reason.) +User=0 + +# Only log timestamps: Gitea keeps each user's own timezone in the database. +Environment=TZ={{ timezone }} + +# The image's VOLUME. Repositories, LFS, the SQLite database, app.ini, sessions, +# logs, and the authorized_keys file the host's sshd reads. +Volume={{ host_directory }}:/data + +# The image starts an sshd of its own on port 22 inside the container. This +# deployment does not use it — git-over-SSH arrives on the *host's* sshd and is +# forwarded with `podman exec` — but it still wants host keys, and it runs as the +# container's root, which under keep-id is a subuid that cannot write into the +# data directory. It gets a scratch directory of its own instead, outside the +# data tree, so nothing it writes can end up in a backup. It is not published, +# and nothing connects to it. +Volume={{ container_sshd_hostkeys }}:/data/ssh + +Network={{ network_name }}.network + +# The image ships no HEALTHCHECK, so the unit defines one and the play gates on +# it. /api/healthz is Gitea's own unauthenticated health endpoint. +HealthCmd={{ health_cmd }} +HealthInterval=30s +HealthTimeout=5s +HealthRetries=3 +HealthStartPeriod=60s + +[Service] +Restart=always +# The first start initialises the SQLite database and runs its migrations before +# the server answers. +TimeoutStartSec=180 + +[Install] +WantedBy=default.target diff --git a/gitea/roles/ssh/tasks/main.yaml b/gitea/roles/ssh/tasks/main.yaml new file mode 100644 index 0000000..7db6e0a --- /dev/null +++ b/gitea/roles/ssh/tasks/main.yaml @@ -0,0 +1,140 @@ +--- +# The host half of Gitea's git-over-SSH support. +# +# Port 22 belongs to the host's sshd and stays that way: clients keep using +# git@git.tanshu.com:user/repo.git, and no remote URL changes. Four host-side +# pieces make that work: +# +# 1. a `{{ git_user }}` login user with no interactive access — every key +# Gitea accepts carries a forced command, so sshd runs the shim and +# nothing else; +# 2. {{ shim_path }}, the shim those forced commands invoke; +# 3. {{ podman_exec_wrapper }} plus a one-command sudoers rule — how the shim, +# running as `{{ git_user }}`, reaches a container owned by +# {{ podman_user }}; +# 4. the sshd and sudoers drop-ins that tie the above together. +# +# What differs from the Docker-era role is the second hop. It used to be an ssh +# into an sshd running *inside* the container; under keep-id that sshd runs as a +# subuid and cannot read the authorized_keys Gitea writes as {{ podman_user }}, +# so the hop is a `podman exec` instead. readme.md, "How git over SSH works +# now", is where the reasoning lives. +# +# Every task here is root-only (users, /usr/local/bin, /etc/ssh, /etc/sudoers), +# so each elevates on its own. The play carries no become, because the Quadlet +# units have to belong to the connection user. + +- name: Ensure the git group exists + ansible.builtin.group: + name: "{{ git_user }}" + become: true + +# The shell matters even though nobody ever gets one: sshd runs a forced command +# through the login user's shell (`/bin/bash -c '...'`), so a nologin shell would +# break git-over-SSH outright. +# +# The uid is deliberately left to the system. On monoco the host git account was +# uid 1003 because the container's git account was 1003 and the two shared one +# uid space; under keep-id the container's 1003 is mapped onto {{ podman_user }} +# instead, so nothing on this host depends on this number any more — not the +# key file (owned by name), not the sudoers rule (matched by name), not the +# authorized_keys hook (run as {{ podman_user }}). +- name: Add the git login user + ansible.builtin.user: + name: "{{ git_user }}" + comment: Gitea SSH login user + group: "{{ git_user }}" + home: "{{ git_home }}" + shell: /bin/bash + create_home: true + become: true + +# There is deliberately no ~/.ssh for this user any more. Its old contents were +# the shim's client key and the authorized_keys Gitea maintains; the key is +# obsolete (nothing authenticates over ssh in this design) and the file lives in +# the container's data tree now, where Gitea writes it. The migration notes in +# readme.md move the file into place. + +- name: Install the git-over-SSH passthrough shim + ansible.builtin.template: + src: gitea.j2 + dest: "{{ shim_path }}" + owner: root + group: root + mode: '0755' + become: true + +- name: Install the container exec wrapper + ansible.builtin.template: + src: gitea-container-exec.j2 + dest: "{{ podman_exec_wrapper }}" + owner: root + group: root + mode: '0755' + become: true + +- name: Install the authorized_keys hook sshd runs for the git user + ansible.builtin.template: + src: gitea-authorized-keys.j2 + dest: "{{ authorized_keys_command }}" + owner: root + group: root + mode: '0755' + become: true + +# valid with `visudo -cf %s`: a syntax error in /etc/sudoers.d does not fail one +# rule, it fails *every* sudo invocation on the host, which on a server reached +# by ssh is a nasty way to find out. +- name: Let the git user reach the container as {{ podman_user }} + ansible.builtin.template: + src: sudoers-gitea.j2 + dest: "{{ sudoers_path }}" + owner: root + group: root + mode: '0440' + validate: /usr/sbin/visudo -cf %s + become: true + +- name: Point sshd at Gitea's authorized_keys + block: + - name: Install the sshd drop-in for the git user + ansible.builtin.template: + src: sshd-gitea.conf.j2 + dest: "{{ sshd_dropin_path }}" + owner: root + group: root + mode: '0644' + register: gitea_sshd_dropin + become: true + + # A drop-in that does not parse is not a failed play, it is a host nobody can + # log in to after the next reboot: sshd reads the whole directory at start-up. + # Validate the resulting configuration before it is ever loaded, and take the + # file away again if it does not parse. + - name: Validate the resulting sshd configuration + ansible.builtin.command: /usr/sbin/sshd -t + changed_when: false + become: true + when: gitea_sshd_dropin.changed + rescue: + - name: Take the sshd drop-in back out + ansible.builtin.file: + path: "{{ sshd_dropin_path }}" + state: absent + become: true + + - name: Fail + ansible.builtin.fail: + msg: >- + {{ sshd_dropin_path }} did not validate, so it has been removed and + sshd is still running — and still restarting — on its previous + configuration. Run `/usr/sbin/sshd -t` on the host to see why. + +# `reload`, not `restart`: a reload re-execs sshd without dropping the connection +# this play is running over. +- name: Reload sshd + ansible.builtin.systemd_service: + name: "{{ sshd_service }}" + state: reloaded + become: true + when: gitea_sshd_dropin.changed diff --git a/gitea/roles/ssh/templates/gitea-authorized-keys.j2 b/gitea/roles/ssh/templates/gitea-authorized-keys.j2 new file mode 100644 index 0000000..a58d60f --- /dev/null +++ b/gitea/roles/ssh/templates/gitea-authorized-keys.j2 @@ -0,0 +1,29 @@ +#!/bin/sh +# sshd's source for the `{{ git_user }}` user's authorized_keys. +# Managed by Ansible (gitea/roles/ssh). Do not edit by hand. +# +# Gitea maintains that file from inside its container, where it runs as +# {{ podman_user }} (container uid {{ gitea_uid }} is mapped onto that user by +# keep-id), so the file belongs to {{ podman_user }} and not to `{{ git_user }}`. +# sshd refuses a file the login user does not own — that is StrictModes — and +# StrictModes cannot be relaxed for one user, because sshd_config(5) does not +# allow the option inside a Match block. So the file is fed to sshd through +# AuthorizedKeysCommand instead: sshd uses whatever this prints, and applies no +# ownership check to it. +# +# Only lines that force the passthrough shim are printed. Gitea's file also +# carries an unforced entry for the "Gitea Host Key" that the Docker-era +# deployment needed so that its shim could ssh into a second sshd inside the +# container. Nothing reads that entry any more, and printing it would hand an +# interactive `{{ git_user }}` login on this host — and therefore rootless +# Podman, through the sudoers drop-in — to anyone holding a copy of that key, +# a backup included. +# +# The pattern is exactly the shape Gitea writes. If it ever changes, +# git-over-SSH stops working while HTTPS keeps serving, and +# podman exec {{ container_name }} cat /data/git/.ssh/authorized_keys +# shows what Gitea writes now. +keys="{{ host_directory }}/git/.ssh/authorized_keys" +[ -r "$keys" ] || exit 0 +grep -E '^command="{{ shim_path }} ' "$keys" +exit 0 diff --git a/gitea/roles/ssh/templates/gitea-container-exec.j2 b/gitea/roles/ssh/templates/gitea-container-exec.j2 new file mode 100644 index 0000000..2980128 --- /dev/null +++ b/gitea/roles/ssh/templates/gitea-container-exec.j2 @@ -0,0 +1,34 @@ +#!/bin/sh +# Runs one authorized_keys command inside Gitea's rootless container. +# Managed by Ansible (gitea/roles/ssh). Do not edit by hand. +# +# This runs as {{ podman_user }} — the user whose rootless Podman owns the +# container, which is the entire reason the wrapper exists. The `{{ git_user }}` +# login user cannot enter another user's rootless container, and there is no +# sshd inside the container to hop through any more (see readme.md, "How git +# over SSH works now"), so the forced command sshd runs for a git client ends up +# here. +# +# $1 SSH_ORIGINAL_COMMAND — the client's original request, passed through so +# Gitea can serve it. +# $@ the rest of the authorized_keys line, i.e. +# /usr/local/bin/gitea --config=/data/gitea/conf/app.ini serv key-N — +# the *container's* Gitea binary, not the host's shim at the same path. +# +# --user, HOME and GITEA_CUSTOM are what sshd used to supply: it ran the command +# as the container's git account, with HOME from /etc/passwd and GITEA_CUSTOM +# from /data/git/.ssh/environment. GITEA_CUSTOM and USER are image defaults and +# come through `podman exec` on their own; HOME does not, and git looks there for +# /data/git/.gitconfig, so it is set explicitly. +# +# XDG_RUNTIME_DIR is set explicitly because `sudo -u` creates no PAM session, so +# Podman would not otherwise find this user's runtime directory — the same reason +# caddy/readme.md shows the `sudo -u … systemctl --user` form with it. +original_command="$1" +shift +exec env XDG_RUNTIME_DIR="/run/user/$(id -u)" \ + podman exec --interactive \ + --user {{ gitea_uid }}:{{ gitea_gid }} \ + --env "HOME=/data/git" \ + --env "SSH_ORIGINAL_COMMAND=$original_command" \ + {{ container_name }} "$@" diff --git a/gitea/roles/ssh/templates/gitea.j2 b/gitea/roles/ssh/templates/gitea.j2 new file mode 100644 index 0000000..cc4e942 --- /dev/null +++ b/gitea/roles/ssh/templates/gitea.j2 @@ -0,0 +1,25 @@ +#!/bin/sh +# Gitea's git-over-SSH passthrough shim — {{ shim_path }} +# Managed by Ansible (gitea/roles/ssh). Do not edit by hand. +# +# The host's sshd runs this, as the `{{ git_user }}` login user, whenever a +# client presents a key whose authorized_keys line is a forced command. Gitea +# writes those lines itself, from inside its container, and each one looks like +# this: +# +# command="/usr/local/bin/gitea --config=/data/gitea/conf/app.ini serv key-5",... +# +# So $@ here is `--config=/data/gitea/conf/app.ini serv key-5`, and $0 is the +# path of Gitea's own binary *inside the container*, which the wrapper resolves +# in there. The path is a contract, not a knob: it is already baked into every +# line of the authorized_keys file copied from monoco. +# +# `{{ git_user }}` cannot reach rootless containers that belong to +# {{ podman_user }}, so the command is handed to {{ podman_exec_wrapper }}, which +# sudo runs as {{ podman_user }}. SSH_ORIGINAL_COMMAND — the client's own +# git-upload-pack / git-receive-pack request — crosses as an argument, because +# sudo strips the environment. +# +# `sudo -n` on purpose: there is no TTY on this path, so a password prompt would +# hang a git push instead of failing it. +exec sudo -n -u {{ podman_user }} {{ podman_exec_wrapper }} "$SSH_ORIGINAL_COMMAND" "$@" diff --git a/gitea/roles/ssh/templates/sshd-gitea.conf.j2 b/gitea/roles/ssh/templates/sshd-gitea.conf.j2 new file mode 100644 index 0000000..62b2f5f --- /dev/null +++ b/gitea/roles/ssh/templates/sshd-gitea.conf.j2 @@ -0,0 +1,20 @@ +# git-over-SSH for Gitea's `{{ git_user }}` login user. +# Managed by Ansible (gitea/roles/ssh). Do not edit by hand. +# +# The authorized_keys file is maintained by Gitea from inside its container, as +# {{ podman_user }}, so `{{ git_user }}` does not own it — which is precisely what +# StrictModes refuses. StrictModes cannot be relaxed for a single user +# (sshd_config(5) does not permit it inside a Match block), so the keys are +# handed to sshd by {{ authorized_keys_command }} instead, which sshd uses +# without any ownership check. +# +# AuthorizedKeysFile=none turns the file lookup off. Without it sshd would still +# read (and log a StrictModes complaint about) ~/.ssh/authorized_keys on every +# connection before falling through to the command. +# +# This drop-in must stay in one file, and the Match block must stay last in it: +# sshd applies everything after a Match to the matched connections only. +Match User {{ git_user }} + AuthorizedKeysFile none + AuthorizedKeysCommand {{ authorized_keys_command }} + AuthorizedKeysCommandUser {{ podman_user }} diff --git a/gitea/roles/ssh/templates/sudoers-gitea.j2 b/gitea/roles/ssh/templates/sudoers-gitea.j2 new file mode 100644 index 0000000..699d728 --- /dev/null +++ b/gitea/roles/ssh/templates/sudoers-gitea.j2 @@ -0,0 +1,17 @@ +# Let Gitea's `{{ git_user }}` login user run exactly one command as +# {{ podman_user }}: the wrapper that execs into Gitea's rootless container. +# Managed by Ansible (gitea/roles/ssh). Do not edit by hand. +# +# This is what replaces the ssh hop the Docker-era deployment used. A rootless +# container belongs to one user and cannot be entered by another, and no sshd +# runs inside the container any more, so the forced command sshd runs for a git +# client has to become `podman exec` — run as {{ podman_user }}. +# +# No arguments are pinned, on purpose. The command line comes from the +# authorized_keys lines Gitea maintains (… serv key-N), and that file is already +# the thing deciding who may reach which repository; restating its format here +# would only add a second place to update whenever Gitea changes it, and pinning +# it loosely enough to be useful (… serv *) would not restrict anything the +# wrapper does not already fix — container, uid and environment are all baked +# into {{ podman_exec_wrapper }}. +{{ git_user }} ALL=({{ podman_user }}) NOPASSWD: {{ podman_exec_wrapper }} diff --git a/gitea/vars/default.yml b/gitea/vars/default.yml index 79944a4..290d7e3 100644 --- a/gitea/vars/default.yml +++ b/gitea/vars/default.yml @@ -1,18 +1,118 @@ --- -http_host: "git.tanshu.com" -http_conf: "git.tanshu.com.conf" -http_url: "https://git.tanshu.com" +# --------------------------------------------------------------------------- +# Rootless Podman / Quadlet +# --------------------------------------------------------------------------- +# Duplicated on purpose from caddy/vars/default.yml: a play cannot include that +# file wholesale, because it also sets `host_directory` — to the Caddy data +# directory — and every playbook defines that for itself. Keep these three in +# sync with caddy/vars/default.yml. (Its fourth, podman_unprivileged_port_start, +# belongs to the podman tasks, which this play does not run: Gitea publishes +# nothing host-side.) +# +# podman_user is derived from the connection: this play carries no become at +# play level, so the Quadlet units belong to whoever Ansible connects as. +# Requires gather_facts (set in playbook.yml). +podman_user: "{{ ansible_facts['user_id'] }}" +podman_data_dir: "{{ ansible_facts['user_dir'] }}/podman-data" +quadlet_dir: "{{ ansible_facts['user_dir'] }}/.config/containers/systemd" + +# --------------------------------------------------------------------------- +# Gitea +# --------------------------------------------------------------------------- +host: git.tanshu.com container_name: gitea -container_image: gitea/gitea:latest +unit_name: "{{ container_name }}.service" -docker_network: "{{ container_name }}_net" -caddy_container: caddy +# Fully qualified on purpose: rootless Podman resolves short names against +# unqualified-search-registries, and there is no TTY to answer a prompt when +# the systemd unit starts. +image: docker.io/gitea/gitea:latest + +# The Quadlet file is .network. caddy/roles/register pins the +# Podman network name to _net (so: gitea.network, gitea_net). +network_name: "{{ container_name }}" + +host_directory: "{{ podman_data_dir }}/{{ container_name }}" + +# Port inside the container that Caddy proxies to — app.ini's HTTP_PORT. Not +# published on the host: Caddy reaches it by container name. +container_port: 3000 + +# The uid/gid of the `git` account the image creates for itself +# (git:x:1003:1003, verified in the image), and therefore the uid Gitea runs as. +# The number is load-bearing three times over: +# +# * UserNS in the Quadlet unit maps it onto podman_user, which is what keeps +# every file Gitea writes owned by podman_user on the host — plain tar and +# rsync backups, no subuids; +# * USER_UID/USER_GID force the entrypoint to keep it, so the file layout and +# the mapping cannot drift apart; +# * roles/ssh execs into the container as exactly this uid. +# +# Change one and the other two have to change with it. +gitea_uid: 1003 +gitea_gid: 1003 + +# Where the image keeps its data *inside* the container, and therefore the +# GITEA_CUSTOM the image's own environment already sets. +gitea_custom: /data/gitea + +# Gitea's own unauthenticated health endpoint. The image ships no HEALTHCHECK, +# so the unit defines one from this and the play gates on it. +health_cmd: "curl -fsS -o /dev/null http://127.0.0.1:{{ container_port }}/api/healthz" + +# Only log timestamps are affected: Gitea stores each user's own timezone. The +# Docker-era container bind-mounted the host's /etc/localtime and /etc/timezone +# instead; TZ is the same intent without depending on either file existing. +timezone: Asia/Kolkata + +# A scratch directory for the sshd *inside* the image, which this deployment +# does not use but which still has to write host keys somewhere. It is kept +# outside the data tree on purpose: the process that runs it is the container's +# root, which under keep-id is a subuid, and a subuid-owned file inside +# {{ host_directory }} would break both a plain tar backup and the play's +# "nothing in here belongs to another user" check. See readme.md. +container_sshd_hostkeys: "{{ podman_data_dir }}/{{ container_name }}-sshd" + +# --------------------------------------------------------------------------- +# The git-over-SSH entry point on the host +# --------------------------------------------------------------------------- +# Clients keep using git@git.tanshu.com:user/repo.git on port 22, served by the +# *host's* sshd, exactly as they did on monoco. No sshd runs inside the +# container: under keep-id the container's root is a subuid that cannot read the +# authorized_keys Gitea writes as podman_user, so the second hop is a `podman +# exec` instead of an ssh. See readme.md, "How git over SSH works now". +# +# `git_home` only has to exist — sshd runs the forced command from it — and its +# uid is deliberately left to the system: nothing in this design pairs the host +# account with the container's uid. See roles/ssh/tasks/main.yaml. +git_user: git +git_home: /home/git + +# The command Gitea writes into every authorized_keys line it maintains. This is +# a contract, not a knob: it is already baked into the lines sitting in the file +# copied from monoco, and renaming it locks every user out of git-over-SSH until +# Gitea rewrites them. See roles/ssh/templates/gitea.j2. +shim_path: /usr/local/bin/gitea + +# Runs as podman_user, via the sudoers drop-in, and does the podman exec. +podman_exec_wrapper: /usr/local/bin/gitea-container-exec + +# What sshd calls to obtain the git user's authorized_keys. +authorized_keys_command: /usr/local/bin/gitea-authorized-keys + +sshd_dropin_path: /etc/ssh/sshd_config.d/60-gitea.conf +sshd_service: ssh +sudoers_path: /etc/sudoers.d/gitea-ssh-passthrough + +# --------------------------------------------------------------------------- +# The shared Caddy unit and its Caddyfile, both owned by caddy/playbook.yml +# --------------------------------------------------------------------------- +caddy_name: caddy +caddyfile_path: "{{ podman_data_dir }}/{{ caddy_name }}/conf/Caddyfile" -caddyfile_path: /var/lib/caddy/conf/Caddyfile snippet_block: | - {{ http_host }} { - reverse_proxy {{ container_name }}:3000 + {{ host }} { + reverse_proxy {{ container_name }}:{{ container_port }} } - -user: 'git'