playbooks/gitea/readme.md

Gitea (rootless Podman + Quadlet)

Runs Gitea on emily as a rootless Podman container managed by a systemd user unit (Quadlet), behind the shared rootless Caddy, with git-over-SSH still arriving on the host's own port 22 so that no client has to change its remote. Replaces the docker_container deployment on monoco.

Two things are worth knowing before anything else:

• This Gitea is SQLite. monoco's live app.ini says DB_TYPE = sqlite3, PATH = /data/gitea/gitea.db, so there is no postgres role here and nothing to restore into the shared cluster. The unused Docker-era Postgres variant that used to sit next to the playbook has been deleted.
• There is no sshd inside the container any more, and the reason is not a preference. Why is the heart of this readme.

Ordering

1. caddy/playbook.yml — the reverse proxy, which owns the Caddyfile this site appends to and is the one play that applies the rootless-Podman prerequisites.
2. The data copy below — before the first run of this playbook. Gitea writes into ~/podman-data/gitea the moment it starts, and a first start on an empty directory serves the install wizard instead of this repository's Gitea.
3. This playbook — run it after the file copy, not before.

postgres/playbook.yml is not part of this ordering. There is no postgres role in this playbook and no database to attach.

Layout

gitea/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── ssh/     # the host half of git-over-SSH: git user, shim, sudoers, sshd drop-in
    ├── gitea/   # the data tree the image cannot create, the Quadlet unit, start + verify
    └── caddy/   # appends this site's block to the shared Caddyfile

On the host	What
~/.config/containers/systemd/gitea.container	the generated unit (Ansible-templated)
~/.config/containers/systemd/gitea.network	the app network, written by caddy/roles/register
~/.config/containers/systemd/caddy.container.d/50-gitea.conf	attaches Caddy to that network
~/podman-data/gitea/	everything Gitea owns: git/ (repositories, LFS), gitea/ (app.ini, gitea.db, sessions, avatars, logs), git/.ssh/authorized_keys
~/podman-data/gitea/ssh/	not Gitea's: a bare mount point, always empty. The in-image sshd's host keys are mounted over it from gitea-sshd below, which hides it from the container — so nothing ever writes here
~/podman-data/gitea-sshd/	scratch space for the image's sshd, which this deployment does not use — see below
/home/git/	the git login user's home; no .ssh — see The pieces on the host
/usr/local/bin/gitea	the passthrough shim sshd's forced command runs
/usr/local/bin/gitea-container-exec	podman exec wrapper, run as the podman user
/usr/local/bin/gitea-authorized-keys	what sshd calls to get the git user's keys
/etc/sudoers.d/gitea-ssh-passthrough	lets git run exactly that one wrapper
/etc/ssh/sshd_config.d/60-gitea.conf	Match User git → the keys hook

How git over SSH works now

What the Docker deployment did — and why it cannot be repeated

client ──ssh git@git.tanshu.com:22──► host sshd (monoco)
                                        │ authenticates against /home/git/.ssh/authorized_keys
                                        │ each line is a forced command:
                                        │ command="/usr/local/bin/gitea --config=/data/gitea/conf/app.ini serv key-5",…
                                        ▼
                              /usr/local/bin/gitea   ← shim, runs as host `git`
                                        │ ssh -p 2222 git@127.0.0.1 "SSH_ORIGINAL_COMMAND=… /usr/local/bin/gitea … serv key-5"
                                        ▼
                              sshd inside the container (published 127.0.0.1:2222)
                                        │ authenticated with the host git user's key
                                        │ — the bare, un-forced "Gitea Host Key" line
                                        ▼
                              container's gitea: `gitea serv key-5`
                                        └─ key-5 → the user, in Gitea's database → serve the repo

That works on monoco only because Docker does not remap uids: container-uid 1003 is host-uid 1003, container-root is host-root, so the app, the sshd and the git login user all share one identity and can read each other's files.

Under rootless Podman with UserNS=keep-id:uid=1003,gid=1003, exactly one container uid is shared with the host — 1003 ↔ tanshu — and everything else is a subuid. Verified on emily from the running postgres unit, which uses the same mechanism:

container 0   → parent 1  → host subuid 165536
container 70  → parent 0  → host tanshu

So the container's own sshd has an impossible job:

Inside the container	On the host	Consequence
Gitea (1003)	tanshu	writes authorized_keys, mode 0600
sshd (0)	subuid 165536	cannot read a 0600 tanshu-owned file

and it cannot be fixed by swapping the mapping: sshd must be root in the container, while Gitea refuses to be root at all — the image aborts with mustNotRunAsRoot() [F] Gitea is not supposed to be run as root. There is no uid that satisfies both, and keep-id can only map one of them onto the host user.

The replacement

The second hop stops being an ssh and becomes a podman exec, run as tanshu, the user who owns the container. The public entry point — the host's sshd on port 22 — is unchanged, and so is every command= line Gitea has already written, because the passthrough shim keeps its path and its arguments:

client ──ssh git@git.tanshu.com:22──► host sshd (emily)
                                        │ keys from gitea-authorized-keys (AuthorizedKeysCommand,
                                        │ because StrictModes refuses a file git does not own)
                                        ▼
                              /usr/local/bin/gitea   ← shim, runs as host `git`
                                        │ sudo -n -u tanshu /usr/local/bin/gitea-container-exec \
                                        │     "$SSH_ORIGINAL_COMMAND" --config=… serv key-5
                                        ▼
                              podman exec -i --user 1003:1003 … gitea \
                                  /usr/local/bin/gitea --config=/data/gitea/conf/app.ini serv key-5
                                        └─ env SSH_ORIGINAL_COMMAND="git-upload-pack '…'"
                                           key-5 → the user → serve the repo

The identity still travels in the command string (key-5), so Gitea's per-user authorisation is exactly what it was. What is gone: the 2222 publish, the container's sshd as a hop, /data/ssh in the data tree, and the "Gitea Host Key" that only existed to authenticate that hop.

Why there is no sshd in the container

• sshd must run as the container's root, which under keep-id is a subuid.
• Gitea refuses to run as root, so it runs as container-uid 1003 = tanshu.
• Gitea maintains authorized_keys itself, as tanshu, mode 0600.
• A subuid cannot read it. StrictModes would also reject it.
• Rewriting the modes is not a fix: sshd ignores world-readable host keys, and the file is rewritten by Gitea after every key change.
• podman exec needs no uid to be shared with the container at all, so it works.

The image still starts an sshd of its own (/etc/s6/openssh), because that is baked into the image. It is unpublished and nothing connects to it, but it does want host keys, and it cannot write them into the data tree for the same subuid reason. It therefore gets ~/podman-data/gitea-sshd, a scratch directory mounted over /data/ssh outside the data tree — so no subuid-owned file ever lands in ~/podman-data/gitea, a plain tar of the data tree stays complete, and the play's "nothing in here belongs to another user" check stays meaningful.

If you prefer no stray directory at all, a tmpfs works in principle: Mount=type=tmpfs,destination=/data/ssh,tmpfs-mode=1777 instead of the Volume= line. Not verified on emily — try it in a scratch copy of the unit and watch podman logs gitea for the image's openssh service coming up.

The pieces on the host

The git login user exists only so sshd has someone to authenticate as. It has /bin/bash because sshd runs a forced command through the login shell (/bin/bash -c '…'); a nologin shell would break git-over-SSH outright. Its uid is deliberately not pinned: on monoco it was 1003 because it had to equal the container's git uid, and under keep-id that pairing no longer exists.

There is no /home/git/.ssh. It used to hold the shim's client key (the "Gitea Host Key", now obsolete — nothing authenticates over ssh any more) and was bind-mounted into the container as /data/git/.ssh. The file that matters now lives in the data tree, where Gitea writes it. The migration below moves it there.

/usr/local/bin/gitea is the shim. Its path is a contract: it is baked into every command= line Gitea has already written, so renaming it locks every user out until Gitea rewrites the file.

/usr/local/bin/gitea-container-exec is the podman exec, with the uid, HOME, GITEA_CUSTOM, SSH_ORIGINAL_COMMAND and XDG_RUNTIME_DIR that the container's sshd used to provide. XDG_RUNTIME_DIR is set explicitly because sudo -u creates no PAM session — the same trick caddy/readme.md shows for sudo -u … systemctl --user.

/etc/sudoers.d/gitea-ssh-passthrough grants git exactly one command as tanshu. It is installed with visudo -cf validation, because a syntax error in /etc/sudoers.d does not break one rule, it breaks every sudo on the host. The trust boundary is worth stating plainly: whoever can obtain a git shell on this host can run containers as tanshu. Nothing gets that shell — every key Gitea accepts carries a forced command — and the hook below is what keeps it that way.

/etc/ssh/sshd_config.d/60-gitea.conf points sshd at gitea-authorized-keys for Match User git. It has to be a hook rather than the plain file because the file belongs to tanshu, which is exactly what StrictModes refuses — and StrictModes cannot be relaxed for one user: OpenSSH rejects it inside a Match block (Directive 'StrictModes' is not allowed within a Match block, confirmed on emily, OpenSSH 10.2p1). AuthorizedKeysCommand output is used without an ownership check, so that is the mechanism. The drop-in is validated with a full sshd -t before sshd is reloaded, and removed again if it does not parse.

/usr/local/bin/gitea-authorized-keys prints Gitea's file, filtered to the lines that force the shim. The unforced "Gitea Host Key" line is deliberately dropped: nothing needs it now, and printing it would hand an interactive git login — and with it rootless Podman, via the sudoers rule — to anyone holding a copy of that key, a backup included.

Who runs as whom

UserNS=keep-id:uid=1003,gid=1003 plus Environment=USER_UID/USER_GID=1003, all three from gitea_uid in vars/default.yml. Gitea therefore writes as container-1003, which is tanshu on the host: everything under ~/podman-data/gitea is tanshu-owned, and plain tar/rsync backups keep working.

User=0 is the fourth line, and without it nothing starts. keep-id does not only map the uid, it also runs the container process as it — so the whole container came up as git, while the image's start-up is written to begin as root: /usr/bin/entrypoint rewrites /etc/passwd, /etc/s6/gitea/setup (sourced by /etc/s6/gitea/run before it su-exec gits the server) prepares /data, and the image's own CMD is s6-svscan /etc/s6, which has to create .s6-svscan/lock inside root-owned /etc/s6. Denied all three, the container exited immediately and Restart=always turned that into a crash loop — which is-active reports as activating, not as a failure. User=0 restores root for the start-up and changes nothing about the mapping above: the server still runs as git through su-exec, which is what satisfies Gitea's own mustNotRunAsRoot check. (nextcloud.container carries the same pair for the same reason.)

The consequence to internalise: container root is a subuid on the host. It holds the usual capabilities inside its user namespace, so the start-up's mkdir, chmod and chown all succeed — and what they create is left owned by that subuid. The image chowns the directories it knows about back to git (chown -R ${USER}:git /data/gitea and /data/git in /etc/s6/gitea/setup), but not everything, so roles/gitea creates the tree up front — gitea/, gitea/conf/, gitea/log/, git/, git/.ssh/, ssh/ — owned by tanshu, with the modes those scripts would have set. Two of them are not housekeeping at all:

• git/.ssh/environment is written by the play rather than left to the setup script, because the setup's copy is the one file under /data it never chowns back to git.
• ssh/ is the mount point for the in-image sshd's host keys. It is hidden inside the container by that mount, so nothing ever chowns it: if Podman has to create it, it lands in the data tree owned by the subuid, and the check at the end of the role fails on it. That is exactly what happened before this was added.

This is the same trap postgres/readme.md documents for PGDATA, arrived at from the other direction.

When it does not start

The container's start-up output is in the journal, and the play prints it for you: every step from Start Gitea through the health check is in one block whose rescue dumps systemctl status, journalctl and podman logs. The lines that have actually shown up, and what each means:

In the journal	Cause
s6-svscan: fatal: unable to open .s6-svscan/lock: Permission denied	the container process is not root — User=0 is missing
sed: can't create temp file '/etc/passwdXXXXXX': Permission denied	same cause: the entrypoint may not rewrite /etc/passwd
DB not initialized / the install page in a browser	the copy from monoco has not run, so this is an empty Gitea

Beyond those, podman logs gitea on a healthy container is clean — no permission errors at all. An earlier draft of this readme listed four chmod … Operation not permitted lines as expected noise; they do not occur, and if you see something like them they are worth reading rather than skimming.

Settings

Variable	Purpose
podman_user, podman_data_dir, quadlet_dir	Rootless-Podman contract, duplicated from caddy/vars/default.yml
container_name	Container, unit and network prefix — gitea
image	docker.io/gitea/gitea:latest, fully qualified on purpose
host_directory	Where all state lives (~/podman-data/gitea)
container_port	3000, the port inside the container Caddy proxies to
gitea_uid / gitea_gid	1003 — the image's own git account. Drives UserNS, USER_UID/USER_GID and the wrapper's --user; see Who runs as whom
gitea_custom	/data/gitea, the image's GITEA_CUSTOM
timezone	TZ for log timestamps only; Gitea stores each user's own timezone
health_cmd	The unit's HealthCmd — /api/healthz, Gitea's unauthenticated probe
container_sshd_hostkeys	Scratch dir for the image's unused sshd, mounted over /data/ssh
git_user, git_home	The SSH login account (git, /home/git)
shim_path	/usr/local/bin/gitea — a contract, not a knob
podman_exec_wrapper, authorized_keys_command	The two helper scripts
sshd_dropin_path, sshd_service, sudoers_path	/etc/ssh/sshd_config.d/60-gitea.conf, ssh, /etc/sudoers.d/gitea-ssh-passthrough
host, caddy_name, caddyfile_path, snippet_block	The Caddy side, as in every service playbook

Keep the three podman_*/quadlet_dir vars in sync with caddy/vars/default.yml — they are duplicated rather than imported because a play cannot include Caddy's vars file wholesale (it also sets host_directory, which every service defines for itself).

Running

ansible-playbook -u tanshu gitea/playbook.yml

The play targets emily directly, so the inventory must contain it — the one in prepare-server/ does, and its ansible.cfg names it.

This play applies only this app's units, data directories and host-side SSH plumbing. The rootless-Podman prerequisites — installing podman, loginctl enable-linger, the low-port sysctl and the Quadlet directory — are the shared task file at caddy/roles/podman/tasks/main.yaml, and are not run here. A host that has already run caddy/playbook.yml has them.

systemctl --user enable gitea is neither needed nor possible: the unit is generated, and Quadlet applies [Install] WantedBy=default.target itself.

Migrating the data from monoco

Everything here is about files Gitea owns. There is no database dump to take: this Gitea is SQLite, and gitea.db travels inside the tree. Run the commands from the machine that holds the monoco and emily SSH aliases.

Two things in the source tree are deliberately not copied:

• /var/lib/gitea/ssh — host keys for the sshd inside the container. That sshd is no longer a hop, nothing connects to it, and the new deployment lets it generate fresh keys in ~/podman-data/gitea-sshd. The Docker-era audit told you to copy them so clients would not see host-key warnings; with the container's sshd out of the path, no client ever sees them.
• /home/git/.ssh/id_ed25519 and friends — the "Gitea Host Key" that authenticated the old second hop. Nothing authenticates over ssh now. Only authorized_keys (and optionally environment) is carried across, into the container's data tree, which is where Gitea expects to maintain it.

# 1. Look before you copy: what is there, and how big is it. `ls -ln` shows
#    numeric owners, because the old container wrote as uid 1003 (`git`).
ssh monoco 'du -sh /var/lib/gitea/*; ls -ln /var/lib/gitea'
ssh monoco 'sudo ls -ln /home/git/.ssh'

# 2. Stop the old container so nothing writes while you copy. SQLite in WAL mode
#    is exactly the thing the backup plan warns about — a live copy can be torn.
#    Leave it stopped: emily is about to become the only writer, and two live
#    copies behind the same hostname would diverge.
ssh monoco 'sudo docker stop gitea'

# 3. Create the destination tree (the playbook would create it too, but the copy
#    needs somewhere to land).
ssh emily 'mkdir -p ~/podman-data/gitea/git/.ssh'

#    If the playbook has already run here — it says so when it does — the app has
#    left a fresh `gitea/conf/app.ini` (and possibly logs) behind, and the digests
#    in step 6 will not match. Clear it first:
# ssh emily 'systemctl --user stop gitea.service; find ~/podman-data/gitea/gitea ~/podman-data/gitea/git -mindepth 1 -delete'

# 4. Copy the data tree, minus the obsolete container-sshd host keys.
#    tar streams it across the workstation; rsync cannot copy remote-to-remote in
#    one command.
#
#    `set -o pipefail` matters: without it the pipeline reports only the
#    *receiving* tar's status, so a read error or a dropped stream on the source
#    side exits 0 and leaves a silently short copy.
#
#    The source needs sudo — the files belong to `git`, not to you.
set -o pipefail
ssh monoco 'sudo tar --exclude="./ssh" -C /var/lib/gitea -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/gitea -xf -'

# 5. Copy the authorized_keys Gitea maintains. It lives outside /var/lib/gitea on
#    the old host, because it was bind-mounted in from the git user's home.
set -o pipefail
ssh monoco 'sudo tar -C /home/git/.ssh -cf - authorized_keys environment' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/gitea/git/.ssh -xf -'

# 6. Verify by content, over both trees, with the same exclusions on each side so
#    the digests are comparable. `.ssh` is excluded here because it is compared
#    on its own in the next step, and `ssh/` never left monoco.
ssh monoco 'cd /var/lib/gitea && sudo find ./git ./gitea ! -path "./git/.ssh*" -type f -print0 \
  | sort -z | xargs -0 -r sudo sha256sum | sha256sum'
ssh emily  'cd ~/podman-data/gitea && find ./git ./gitea ! -path "./git/.ssh*" -type f -print0 \
  | sort -z | xargs -0 -r sha256sum | sha256sum'
# identical digests = a byte-faithful copy

ssh monoco 'sudo sha256sum /home/git/.ssh/authorized_keys | cut -d" " -f1'
ssh emily  'sha256sum ~/podman-data/gitea/git/.ssh/authorized_keys | cut -d" " -f1'

# 7. Nothing may end up owned by root or by a subuid, and sshd's hook must be
#    able to read the file it is about to be handed.
ssh emily 'find ~/podman-data/gitea ! -user tanshu'          # empty = correct
ssh emily 'ls -ln ~/podman-data/gitea/git/.ssh/authorized_keys'

# 8. Deploy — after the copy, so Gitea finds the real data on first start.
cd prepare-server && ansible-playbook -u tanshu ../gitea/playbook.yml

If the link is slow or drops, do it in two hops instead: resumable, and it shows progress.

rsync -aP --exclude='ssh' monoco:/var/lib/gitea/ ~/gitea-copy/
rsync -aP ~/gitea-copy/ emily:~/podman-data/gitea/
rsync -aP monoco:'/home/git/.ssh/{authorized_keys,environment}' ~/gitea-ssh-copy/
rsync -aP ~/gitea-ssh-copy/ emily:~/podman-data/gitea/git/.ssh/

Rolling back means rolling back all of it: ssh monoco 'sudo docker start gitea' is only meaningful while monoco's data is still intact, and it would then be writing to a gitea.db that emily's copy has already moved past. Never run both at once.

Verification

systemctl --user status gitea
journalctl --user -u gitea -f
podman logs gitea                      # clean — see "When it does not start" above

# What the playbook checks for you:
podman healthcheck run gitea           # /api/healthz
find ~/podman-data/gitea ! -user tanshu   # empty

# The unit is where it should be, and the image's sshd has its scratch space:
podman inspect gitea --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'

# The host's SSH half, piece by piece:
sudo sshd -T -C user=git | grep -i authorizedkeys     # the hook, and only for git
sudo -u tanshu /usr/local/bin/gitea-authorized-keys | head -3
sudo -u tanshu /usr/local/bin/gitea-authorized-keys | grep -c '^ssh-'   # 0 — the bare host-key line is dropped
sudo visudo -cf /etc/sudoers.d/gitea-ssh-passthrough
# the wrapper reaches the container as the right user:
sudo -u tanshu /usr/local/bin/gitea-container-exec "" /usr/local/bin/gitea --version
# and the git user is allowed to run exactly that wrapper:
sudo -u git sudo -n -u tanshu /usr/local/bin/gitea-container-exec "" /usr/local/bin/gitea --version

# The site, end to end.
curl -sSI https://git.tanshu.com

Then, from a client whose key is registered in Gitea:

ssh -T git@git.tanshu.com                     # "Hi <user>! You've successfully authenticated…"
git clone git@git.tanshu.com:tanshu/playbooks.git

and push to a repository you own — that is the check that per-user authorisation still works, because the identity travels in the serv key-N argument and not in the shim's own credentials. Then exercise the web UI and confirm a repository list renders, which is the check that gitea.db came across.

Finally, reboot the host and confirm the unit returns unattended — without loginctl enable-linger tanshu it will not, and the failure is silent.

Notes and gotchas

• DISABLE_SSH must stay false, and Gitea's built-in SSH server must stay off (START_SSH_SERVER absent or false) in app.ini. Gitea only maintains authorized_keys for an external sshd — which the host's sshd is here. Turn the built-in server on and Gitea stops writing the file the host's sshd reads, and git-over-SSH stops working with no error anywhere obvious. The copied app.ini already has this right.
• SSH_PORT = 22 stays as it is. It is what makes Gitea offer git@git.tanshu.com:user/repo.git rather than a URL with a port in it, and the host's sshd is what answers on 22. SSH_LISTEN_PORT = 22 refers to the container's own sshd port, which this deployment does not use.
• The git user can reach rootless Podman as tanshu through one sudoers rule. That is the price of a two-user design and it is bounded to one root-owned wrapper; see The pieces on the host.
• The authorized_keys filter is a format dependency. If Gitea ever stops writing command="/usr/local/bin/gitea … lines, the hook prints nothing and git-over-SSH breaks while HTTPS keeps serving. Diagnose with sudo -u tanshu /usr/local/bin/gitea-authorized-keys and podman exec gitea cat /data/git/.ssh/authorized_keys.
• Pushing over SSH runs no hooks through the host user. Hooks execute inside the container, as git = tanshu, so a repository hook can write anywhere tanshu can. Same as before the move, but worth knowing before enabling server-side hooks.
• latest floats. Neither the old playbook nor this one pins a digest, so an image bump is a normal podman pull away and may run new database migrations on first start.
• The old backup script's docker exec -u git gitea gitea dump … needs rewriting for this deployment: sudo podman exec from root's crontab cannot enter another user's rootless containers, so anything scheduled belongs in a systemd user timer next to the Quadlet units. See the Backups section of postgres/readme.md for the shape of it.
• DNS. git.tanshu.com must resolve to emily before Caddy can obtain a certificate for it.
• Do not restart the container on monoco while emily is serving. It would write to the old data tree, which is being kept intact purely as a rollback.

Not done here

• Rewriting the backup script for rootless Podman and moving it to a systemd user timer (above).
• Rotating the secrets that live in app.ini in plaintext — SECRET_KEY, INTERNAL_TOKEN, LFS_JWT_SECRET, JWT_SECRET — and the fleet-wide 123456-style passwords elsewhere in this repo.
• Templating app.ini. Gitea's own configuration travels with the data, and it carries those secrets; a fresh instance with no data gets the image's install wizard instead, which is a reasonable outcome but not a supported path here.