Add facility to borg-backup role to run a command and save the output
of it to a separate archive file during the backup process.
This is mostly useful for database backups. Compressed on-disk logs
are terrible for differential backups because revisions have
essentially no common data. By saving the uncompressed stream
directly from mysqldump, we allow borg the chance to de-duplicate,
saving considerable space on the backup servers.
This is implemented for our ansible-managed servers currently doing
dumps. We also add it to the testinfra.
This also separates the archive names for the filesystem and stream
backup with unique prefixes so they can be pruned separately.
Otherwise we end up keeping only one of the stream or filesystem
backups which isn't the intention. However, due to issues with
--append-only mode we are not issuing prune commands at this time.
Note the updated dump commands are updated slightly, particularly with
"--skip-extended-insert" which was suggested by mordred and
significantly improves incremental diff-ability by being slightly more
verbose but keeping much more of the output stable across dumps.
Change-Id: I500062c1c52c74a567621df9aaa716de804ffae7
The database dumps are ~3gb each, so are starting to fill up the disk.
Reduce on-disk rotations to 2 and rely on remote backups more.
Change-Id: Ie8fc8fd7c4c36ca37e6813104f4c711ea55186ab
We're using logrotate to keep a small number of db backups locally. We
write these backups to disk compressed. We don't want logrotate to
recompress them. This is unnecessary extra work.
Change-Id: Iafe1628ff421f47cf3e5cbee14998eeceb60be4c
In places like crontab entries we use full paths to executables because
PATH is different under cron. Unfortunately, this meant we broke
docker-compose commands using /usr/bin/docker-compose when we installed
it under /usr/local/bin/docker-compose. In particular this impacted
database backups on gitea nodes and etherpad.
Update these paths so that everything is happy again.
Change-Id: Ib001baab419325ef1a43ac8e3364e755a6655617
We want to use stop_grace_period to manage gerrit service stops. This
feature was added in docker-compose 1.10 but the distro provides 1.5.
Work around this by installing docker-compose from pypi.
This seems like a useful feature and we want to manage docker-compose
the same way globally so move docker-compose installation into the
install-docker role.
New docker-compose has slightly different output that we must check for
in the gitea start/stop machinery. We also need to check for different
container name formatting in our test cases. We should pause here and
consider if this has any upgrade implications for our existing services.
Change-Id: Ia8249a2b84a2ef167ee4ffd66d7a7e7cff8e21fb
We need to install the ep_headings plugin. We've got a hack in place
to do it in prod, but it seems like every time we npm install the
plugin in ansible, we need to restart the etherpad container
for $reasons.
It's cleaner to just build an image based on upstream with ep_headings
installed.
Change-Id: Id3b31bbabbbc5f6dcfe56486e48916384292f693
We need to install the ep_headings plugin, which we were doing
before. We could also do this by building a custom image, but for
now just bind-mount it in.
Change-Id: I95f7c0dff4f1de8709a1e5e32eb818ef2852a654
Upstream likes building the settings file into the image, but that's
less exciting, let's bind-mount ours in.
Depends-On: https://review.opendev.org/717491/
Change-Id: Ia1894d884ef2a84e1282345b77fe07bf8898f367
