The buster-backports suite on Debian mirrors is not signed by the
old jessie signing key we have set to verify in reprepro, but also
we're not mirroring Debian 8/jessie any longer anyway. Replace that
list with the 9/stretch and 10/buster signing keys and switch to
longer key IDs which match the names used for them in the Puppet
manifest. Also add Puppetry and keyfile for the buster keys so that
they will be installed accordingly. The official list of keys can be
found here: https://ftp-master.debian.org/keys.html
Change-Id: Ia193f040b2b707329948955eb091a186eabf8096
Add the gitea07.opendev.org and gitea08.opendev.org servers into the
haproxy pools now that they've been seeded with current data. Remove
the create repos task disable list entries for them as well.
Change-Id: I69390e6a32b01cc1713839f326fa930c376282af
Add new IP addresses to inventory for the rebuilds, but don't
reactivate them in the haproxy pools yet (they're already excluded
from the repository creation task).
Change-Id: I1e3fc1ba56015eeab2c6256b3f90188ecabf23cc
Add the gitea05.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 05 to 07 and 08, and remove 07 and 08 from the
Ansible inventory and comment them out in the haproxy pools in
preparation for replacement.
To the casual observer it may appear gitea06 is being skipped, but
it was replaced first out of sequence due to filesystem corruption
during the PTG. The increased performance of the 75% of the nodes
which have already been replaced means we can get by doing the final
25% at the same time (so two servers at once).
Change-Id: Ia49157c16582b7ed0dbef3eb9d07bf7f1d4450b9
The fortnebula mirror is being rebuilt while the environment there
is under some refactoring. The old mirror isn't reachable any longer
so removing it from our inventory while adding its replacement
should be safe.
Also update the letsencrypt playbooks for the new name.
Change-Id: I789248e4216f4cf059ccc5b071c2a784f9c629e9
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).
Change-Id: I36c188992f4787d4e7c5c952eac5fb0bbdc5a627
Add the gitea04.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 04 to 05, and remove 05 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.
Change-Id: I4cd1fef399e527771a26efee8a39952694f3ce6b
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).
Change-Id: I8b43c6f9cb41452c7f64862a2b401dc0d1b7ef3d
Add the gitea03.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 03 to 04, and remove 04 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.
Change-Id: Id5817f8265996862a7e0810b9fb9e3d78be5d066
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).
Change-Id: Id4076e179bee82b03822f59803865eaa60118334
Add the gitea02.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 02 to 03, and remove 03 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.
Change-Id: I4b51291311064c60d4bb2d90bec6e5cb90a54f3c
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet. Also switch the exclusion
for 01 to 02 for the repository creation task.
Change-Id: I6c4a437316627a723e6bb6c15fdce86a5e847042
The global inventory is used when launching nodes so if we want to
replace a server we have to remove it from the inventory first. This is
that step for replacing gitea02.
Note that when adding it back for the new server there are some edits to
make to the playbooks as noted in the gitea sysadmin docs.
We do also remove this instance from haproxy as well to prevent unwanted
connections while we flip things over.
Change-Id: I53a3f517d46d046cb59e3185ca19ba3df55d8466
Now that the replacement gitea01 server has up to date content, add
it back to the haproxy configuration.
Change-Id: I24b4659603efa1861fed1238b8eda6c3f6c11a14
The install-docker role uses the apt-key utility which expects to
have GPG installed, so include the package for it (this seems to
have been manually installed or preinstalled on the images for our
existing Gitea servers, but our new images do not include it).
Change-Id: I28d748fab35e22219a7278603ed984aaa7658ef0
This rsync'd mirror is now being managed by the opendev mirror update
server. Remove it from the older openstack server to avoid a conflict in
excludes around sclo repo.
Currently we have opendev adding sclo and openstack removing it.
Change-Id: I599ee7d0fab8c5e2a060aff86bce20f1f8d4f54b
To provide a stepwise upgrade path from 2.13 running directly to
2.15 in a container, make a container image containing the war we're
using currently. This should let us make a change to how we run the
war without changing the war at all, and then update the war.
Instead of trying to make a clean build for gerrit 2.13 inside of a
builder image, just have it wget the already built wars and jars we
have.
There are pieces of this that duplicate what's being done in puppet,
but in this context it's not immediately clear these are important to
do. However, it's also not clear they're a bad idea.
The gerrit 2.15 build needs a newer bazel. Looking at the CI scripts
that are used by gerrithub, we find that they use bazel 0.26.1
and nodesource v10. Use the bazel image published by google to get
a bazel builder image.
Set gerrit uid/git to 3000 in both images to match the existing
directory ownership so that bindmounting doesn't face permissions
problems.
Change-Id: I3533f01c0859ed50640dcfd98023994c5867c056
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet.
Note this switches the gitea testing to use a host called gitea99 so
that it doesn't conflict with our changes of the production hosts.
Change-Id: I9779e16cca423bcf514dd3a8d9f14e91d43f1ca3
The gitea role will restart gitea if images have updated. We'd like
to not stop them all at the same time. Do serial: 1 so that we update
one backend at a time.
Change-Id: I5ce7f6d8d25a1cf7ddbe901ec6b91860ceaf5bd1
Add an Ansible handler to send a hangup signal through
docker-compose to the running haproxy daemon any time the task to
update its configuration fires.
Change-Id: I1946c1e7eaaa8a8e2209007b5d065dba952ec6e2
This adds the simplest form of health checking to haproxy, a tcp check
to the backends. We can do more sophisticated checks like checking ssl
negotiates or even HTTP requests but for now this is probably a good
improvement.
Change-Id: I3c6b07df4b3e0c380c757e1e5cb51ae0be655f34
Zuul has hit a scenario where a git repo update was unable to talk to
gerrit via ssh because it had reached its per user connection limit [0].
This then led to some openstack job failing [1].
The default limit (which we were using) is 64 connection per user.
Apparently this is not quite enough for a busy zuul? Increase this by
50% up to 96.
[0] http://paste.openstack.org/show/754741/
[1] http://lists.openstack.org/pipermail/release-job-failures/2019-July/001193.html
Change-Id: Ibeca2208485608f3b61aa716184165342bfcc3c9
We ended up running into a problem with nodepool built control plane
images (has to do with boot from volume not allowing us to delete images
that are in use by a nova instance). We have decided to clean this up
and go back to not doing this until we can do it more properly.
Note this isn't a revert because having a group for access to control
plane clouds does seem like a good idea in general and I believe there
have been changes we'd have to resolve in the clouds.yaml files anyway.
Depends-On: https://review.opendev.org/#/c/665012/
Change-Id: I5e72928ec2dec37afa9c8567eff30eb6e9c04f1d
The global inventory is used when launching nodes so if we want to
replace a server we have to remove it from the inventory first. This is
that step for replacing gitea01.
Note that when adding it back for the new server there are some edits to
make to the playbooks as noted in the gitea sysadmin docs.
We do also remove this instance from haproxy as well to prevent unwanted
connections while we flip things over.
Change-Id: If32405b1302353f1f262a30b7392533f86fec1e4
The OpenStack/OpenDev PPA repositories are currently undocumented.
Add some information on where to find things.
Change-Id: Iea03c5d558b3dd6af9f7c860dfcc75a71dc59d9f
Haproxy wants to log to syslog (and not stdout for performance reasons,
see https://github.com/dockerfile/haproxy/issues/3). However there is no
running syslog in our haproxy container. What we can do is mount in the
host's /dev/log and have haproxy write to the hosts syslog to get
logging.
Do this via a docker compose volume bind mount.
Change-Id: Icf4a91c2bc5f5dbb0bfb9d36e7ec0210c6dc4e90
We are booting instances outside of rax and they don't always come with
extra devices that can be repurposed for swap. If in that case then
create a swapfile instead.
Note we do not use fallocate as swapon's manpage says this is suboptimal
with the linux kernel's swap implementation.
Change-Id: I8b9ce18c18e4069aba7de27bb6a9927627b15b49
We're making these requests to localhost over an ssh connection.
The password warning, on the other hand, is a real thing. Let's not
log the gitea password when we run this in prod.
Change-Id: I2157e4027dce5ab9ebceb3f78dbeff22a83d9fad
