The mirror-update server uses /var/run/reprepro to stash reprepro flock
files. We do that to ensure that we don't have stale locks after a
reboot bceause /var/run is cleaned on reboot. Problem is we rely on
daily ansible runs to recreate this dir which means that after a reboot
we can wait up to 24 hours before we get reprepro mirroring again.
Fix this via the use of tmpfiles.d which instructs systemd to create the
dir for us on boot. We specifically note (via the !) that this directory
should only be created on boot and we set the age value to - to prevent
systemd from deleting this directory.
Change-Id: I68e49475c54e756ce5a6933390dbe13ace976c29
For reasons explained in [1] Debian's lsb_release.py on bullseye is
falling back to probing "apt-cache policy"
When (as currently), stretch is the testing release,
/etc/debian_version contains "stretch/sid", as shipped by
base-files. It is therefore impossible to rely on that file to
differentiate between a host running testing or unstable without
asking apt what is actually preferred when installing packages
(through parsing `apt-cache policy`). That's how `lsb-release --
codename` returns "sid" _xor_ "stretch".
The problem is, this parses the output of "apt-cache policy" which
fails for two reasons; firsly we have cleared out all the cache files,
so our hosts return anything until "apt-get update" is run, but
secondly because our mirrors do not have a "label" that matches in
this code at [2]
e.g. what we get out of "apt-cache policy" is
500 https://mirror.dfw.rax.opendev.org/debian bullseye/main amd64 Packages
release o=Debian,n=bullseye,c=main,b=amd64
origin mirror.dfw.rax.opendev.org
which is missing a "l=" field to make this parsing recognise it as a
valid source.
The label is set by reprepro [3]
Label
This optional field is simply copied into the Release files.
Add a label to make our mirrors look more like regular mirrors.
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845651
[2] https://sources.debian.org/src/lsb/11.1.0/lsb_release.py/#L191
[3] https://manpages.debian.org/stretch/reprepro/reprepro.1.en.html
Change-Id: Id705acbb3a01f43ae635a24fa3c24d0a05bdaa16
Starting in bullseye, Debian's security suite will add -security to
dist codenames, meaning we have stretch, buster, and
bullseye-security entries. Looks inconsistent, but is actually
correct.
Change-Id: I34806145f099868c2cdd95893b69cb1f4915f56f
Call `reprepro export` to always recreate indices, even for empty
dists. This is sort of a shotgun approach, local testing on the
server indicates it increases total time of a noop update by ~5.5
minutes for the "debian" repo, which is by far the worst case of
anything we mirror.
If this proves problematic, we can engineer a more targeted solution
to check for empty dists and only export those.
Change-Id: I7e39e427e1941f055fae0408e4c1f2a2f2b35547
It seems we're trying to rotate everything in this directory, leading
to an ever increasing set of weird rotation files as old rotations get
rotated themselves.
Change-Id: Ifd53879061baac35253782126367016b74a9cb70
The config should use the full path to the config directory, append
/etc/reprepro to the job.
Currently all the reprepro jobs hash to the same start time because it
uses the hostname as a seed. Use the unique string name as the seed
so each job starts at a unique time.
Change-Id: If2745d0cd274f390dbff6337b7a44093b5919908
This converts the reprepro mirror script to use the common functions
for timestamps and vos release. This function ssh's to the AFS server
and runs vos release directly there, avoiding many issues with
kerberos timeouts. This has been working successfully for the rsync
mirrors. This will also send stats back so we can keep an eye on the
timing.
Change-Id: I1be29f2d9ecaad03b22c87819e5ae8d16c4f177e
Enable the Ansible based cron jobs, and disable the puppet host
versions to cut over the mirroring to the new server.
Change-Id: I0ffb1c484e64e67f5a5017dc3c3c8ebcdc3845c8
I missed these in the original port. For some reason we are
installing these directly from upstream keyservers in the puppet,
rather than from files like everything else.
Change-Id: Ie1fa956b96f3e6d091b3ffcaab5e0be370da8fc7
In converting this to ansible I forgot to install the reprepro keytab.
The encoded secret has been added for production.
Change-Id: I39d586e375ad96136cc151a7aed6f4cd5365f3c7
This converts the reprepro configuration from our existing puppet to
Ansible.
This takes a more direct approach; the templating done by the puppet
version started simple but over the years grew several different
options to handle various use-cases. This means you not only had to
understand the rather obscure reprepro configuration, but then *also*
figure out how to translate that from our puppet template layers.
Here the configuration files are kept directly (they were copied from
the existing mirror-update.openstack.org) and deployed with some light
wrapper tasks in reprepro/tasks/utils which avoids most duplication.
Note the initial cron jobs are left disabled so we can run some manual
testing before letting it go automatically.
Change-Id: I96a9ff1efbf51c4164621028b7a3a1e2e1077d5c
