Previously we were doing this weekly. Gerrit does this daily. "Split"
the difference and do gitea every other day.
We have noticed that replication to gitea can be slow at times. One idea
is that the less packed repos on the gitea side may make negotiating the
updates slower. Pack more often to see if this helps.
Change-Id: I8961007dce3e448bfdbf1c5f3e8dfc5ec8eb82fb
We now depend on the reverse proxy not only for abuse mitigation but
also for serving .well-known files with specific CORS headers. To
reduce complexity and avoid traps in the future, make it non-optional.
Change-Id: I54760cb0907483eee6dd9707bfda88b205fa0fed
This makes a number of changes and in depth testing is probably
warranted.
* Bump the golang version to 1.16 to match upstream's dockerfile
golang version.
* Bump the nodejs version to latest stable which is v16.x and
consume it from the buster repo since our image is based on buster.
* Bump the gitea version to 1.14.4.
* Rename all of the opendev-.*.png logo files to logo-.*.png as the
names of these files are no longer customizable. The reason for this
is the user settable json manifest has been removed.
* We keep opendev-sm.png because Gerrit apparently loads this?
* Update html templates to be in line with 1.14.4's templates.
* Use the gitea `admin user create` command instead of `admin
create-user`. While I can't find removal or deprecation notices for
the previous command the current docs note you should use the new
version and the old one is failing in CI.
Change-Id: I0a05ebb963cca5be88aeb2f859bfbeefa0f9c8e0
When we added Apache as a filtering proxy on our Gitea backends in
order to more easily mitigate resource starvation, we did not set
any tuning to tell it when to recycle worker processes. As a result,
backends may continue serving requests with workers which pre-date
certificate rotation. This problem has also become more broadly
prevalent throughout our services with the introduction of Let's
Encrypt's 3-month certificate expirations as compared to our
previous 2-year certificates.
Add the same MaxConnectionsPerChild tuning to our Gitea backend
proxies as we use for our static sites and mirror servers.
Change-Id: I77d89385178a30f7dc5d04bedd0ab3772865c09f
I introduced this typo with I500062c1c52c74a567621df9aaa716de804ffae7.
Luckily Ibb63f19817782c25a5929781b0f6342fe4c82cf0 has alerted us to
this problem.
Change-Id: I02bf2f4fa1041642a719100e9591bf5cd1a0bf49
Add facility to borg-backup role to run a command and save the output
of it to a separate archive file during the backup process.
This is mostly useful for database backups. Compressed on-disk logs
are terrible for differential backups because revisions have
essentially no common data. By saving the uncompressed stream
directly from mysqldump, we allow borg the chance to de-duplicate,
saving considerable space on the backup servers.
This is implemented for our ansible-managed servers currently doing
dumps. We also add it to the testinfra.
This also separates the archive names for the filesystem and stream
backup with unique prefixes so they can be pruned separately.
Otherwise we end up keeping only one of the stream or filesystem
backups which isn't the intention. However, due to issues with
--append-only mode we are not issuing prune commands at this time.
Note the updated dump commands are updated slightly, particularly with
"--skip-extended-insert" which was suggested by mordred and
significantly improves incremental diff-ability by being slightly more
verbose but keeping much more of the output stable across dumps.
Change-Id: I500062c1c52c74a567621df9aaa716de804ffae7
We're using logrotate to keep a small number of db backups locally. We
write these backups to disk compressed. We don't want logrotate to
recompress them. This is unnecessary extra work.
Change-Id: Iafe1628ff421f47cf3e5cbee14998eeceb60be4c
This adds an option to have an Apache based reverse proxy on port 3081
forwarding to 3000. The idea is that we can use some of the Apache
filtering rules to reject certain traffic if/when required.
It is off by default, but tested in the gate.
Change-Id: Ie34772878d9fb239a5f69f2d7b993cc1f2142930
In places like crontab entries we use full paths to executables because
PATH is different under cron. Unfortunately, this meant we broke
docker-compose commands using /usr/bin/docker-compose when we installed
it under /usr/local/bin/docker-compose. In particular this impacted
database backups on gitea nodes and etherpad.
Update these paths so that everything is happy again.
Change-Id: Ib001baab419325ef1a43ac8e3364e755a6655617
We want to use stop_grace_period to manage gerrit service stops. This
feature was added in docker-compose 1.10 but the distro provides 1.5.
Work around this by installing docker-compose from pypi.
This seems like a useful feature and we want to manage docker-compose
the same way globally so move docker-compose installation into the
install-docker role.
New docker-compose has slightly different output that we must check for
in the gitea start/stop machinery. We also need to check for different
container name formatting in our test cases. We should pause here and
consider if this has any upgrade implications for our existing services.
Change-Id: Ia8249a2b84a2ef167ee4ffd66d7a7e7cff8e21fb
Gerrit replication plugin is good about retrying replication if its
connectivity to the remote fails. It however thinks everything is happy
if it can connect and push even when gitea-web isn't running.
Make the whole replication system happier by stopping gitea-ssh before
other services and starting it after other services. This way gerrit
should fail to replicate until gitea is ready for it to ssh in.
Change-Id: I3440d8dd8a01a3aaf5d18c9c2ca48e7ead63856f
1.10 introduces a PASSWORD_COMPLEXITY setting with a default value
of lower,upper,digit,spec - which requires passwords to have an
upper, lower, digit and special character. Our example password does
not have this, so set the PASSWORD_COMPLEXITY setting. We could
alternately leave it at the default and ensure that our passwords
meet the spec.
The sshd_config file is templated now, so we can set the listen port
via env var.
Change-Id: I6e4b595eabb9c6885d78fff1109ea9f602e89ef7
This is the first step in managing the opendev.org cert with LE. We
modify gitea01.opendev.org only to request the cert so that if this
breaks the other 7 giteas can continue to serve opendev.org. When we are
happy with the results we can merge the followup change to update the
other 7 giteas.
Depends-On: https://review.opendev.org/694182
Change-Id: I9587b8c2896975aa0148cc3d9b37f325a0be8970
Randomising the time of this job should help avoid a thundering herd
of I/O intensive operations in the gitea environment.
Change-Id: I035f7781a397665357b6d039b989ab9fe6a46b8a
Add the full remote_puppet_git playbook that we actually use in
production so that we can test the whole kit and caboodle. For
now don't add a review.o.o server to the mix, because we aren't
testing anything about it.
Change-Id: If1112a363e96148c06f8edf1e3adeaa45fc7271c
Sadly, as readable as the use of the uri module to do the interactions
with gitea is, more reent ansible changed how subprocesses are forked
and this makes iterating over all the projects in projects.yaml take
an incredibly long amount of time.
Instead of doing it in yaml, make a python module that takes the list
one time and does looping and requests calls. This should make it be
possible to run the actual gitea creation playbook in integration tests.
Change-Id: Ifff3291c1092e6df09ae339c9e7dddb5ee692685
During a db recovery to rebuild a host using the existing db backups
resulted in a corrupt mysql.proc table. The issue seemed to be
attempting to restore the mysql database. Instead of dumping all
databases lets just backup the one we care about: gitea.
Change-Id: Ia2c87b62736fda1c8a9ce77126e383ec74990b4a
The stdout progress feed from `git gc` is fairly verbose and
targeted at audiences running it interactively. Since our cron for
this iterates over thoudands of repositories on our Gitea servers,
we don't need to send the progress info to all our sysadmins by
E-mail. Instead use the --quiet option to the gc subcommand so that
progress output will be suppressed.
If this still proves too verbose (as in, continues to result in
E-mail to root even when there are no failures), we can try
redirecting stdout to /dev/null.
Change-Id: Idc06e48cbf85e127a343c2a3cf51a35e6ed09685
This isn't added as a separate role because it heavily relies on the
gitea deployment specific (docker-compose, service names, etc). If we
end up running more services with docker-compose and databases we can
probably make this reconsumable.
Change-Id: I7b9084a8a90a86f73f5b24de505978d3f286850b
As new change refs accumulate, replication pushes and page loads
will take longer as git stats all of the refs/ files. To avoid
that, pack refs and gc every week to keep the number of files
and space used minimal.
Change-Id: Iff273ebbc25a512ab7e12b8418ceb30e7c722f92
This is a first step toward making smaller playbooks which can be
run by Zuul in CD.
Zuul should be able to handle missing projects now, so remove it
from the puppet_git playbook and into puppet.
Make the base playbook be merely the base roles.
Make service playbooks for each service.
Remove the run-docker job because it's covered by service jobs.
Stop testing that puppet is installed in testinfra. It's accidentally
working due to the selection of non-puppeted hosts only being on
bionic nodes and not installing puppet on bionic. Instead, we can now
rely on actually *running* puppet when it's important, such as in the
eavesdrop job. Also remove the installation of puppet on the nodes in
the base job, since it's only useful to test that a synthetic test
of installing puppet on nodes we don't use works.
Don't run remote_puppet_git on gitea for now - it's too slow. A
followup patch will rework gitea project creation to not take hours.
Change-Id: Ibb78341c2c6be28005cea73542e829d8f7cfab08
This ensures that we cleanup images that are superceded and no longer
necessary. We do this to avoid filling the disk with docker images.
Note that we use the -f flag to avoid being prompted by docker image
prune for confirmation.
Change-Id: I8eb5bb97d8c66755e695498707220c9e6e7b2de0
It's possible that we're not allowing long enough time for mariadb
to stop cleanly. https://github.com/docker-library/mariadb/issues/201
indicates that adding a stop time might be useful. The default is 10,
bump it to 60.
Change-Id: Id7a815d1508fe6d8f79818c9109cbf89533bb2a6
This will make sure that the latest relevant images are in the
local image storage, and therefore, will cause docker-compose up
to recreate containers when the images are updated.
Change-Id: Ic6f0bc8c8aea5b5c16501f4ab5d3095fb70c0ff7
