Previously we were only managing root's known_hosts via ansible but even
then this wasn't happening because the gerrit_self_hostkey var wasn't
set anywhere. On top of that we need to manage multiple known_hosts
because gerrit must recognize itself and all of the gitea servers.
Update the code to take a dict of host key values and add each entry to
known_hosts for both the root and gerrit2 user.
We remove keyscans from tests to ensure that this update is actually
working.
Change-Id: If64c34322f64c1fb63bf2ebdcc04355fff6ebba2
With our system-config-run gerrit/review jobs we have much less need
for a dedicated server to stage changes on. Remove in prepartion of
server cleanup.
Change-Id: I9430f7a2432324a184e3a4f7e41f9e5150c0200c
Start backing up the new review server. Stop backing up the old
server. Fix the group matching test for the new server.
Change-Id: I8d84b80099d5c4ff7630aca9df312eb388665b86
This moves review02 out of the review-staging group and into the main
review group. At this point, review01.openstack.org is inactive so we
can remove all references to openstack.org from the groups. We update
the system-config job to run against a focal production server, and
remove the unneeded rsync setup used to move data.
This additionally enables replication; this should be a no-op when
applied as part of the transition process is to manually apply this,
so that DNS setup can pull zone changes from opendev.org.
It also switches to the mysql connector, as noted inline we found some
issues with mariadb.
Note backups follow in a separate step to avoid doing too much at
once, hence dropping the backup group from the testing list.
Change-Id: I7ee3e3051ea8f3237fd5f6bf1dcc3e5996c16d10
Let's avoid changing this and breaking Depends-On again by adding an
explicit warning to the code that sets the config.
Change-Id: Idcb77d8b0b53c56ea7f15f18e001f8bc9a001c98
Talking to review01.o.o in the Zuul gerrit connection config broke
depends-on handling as the urls would all need to be
https://review01.opendev.org/123456 and then later
https://review02.opendev.org/123456 but people use
https://review.opendev.org/123456.
This change was made to simplify DNS updates during the gerrit server
move but we should be able to handle those via manual landing of changes
and running of playbooks instead. Partially revert
e05257e1b7b70b18cb7b1349278e2c786a565512 to fix the depends-on handling.
Change-Id: Ie628b2627c263d88e466205af2a3d0418d6df7d3
The paste service needs an upgrade; since others have created a
lodgeit container it seems worth us keeping the service going if only
to maintain the historical corpus of pastes.
This adds the ansible to deploy lodgeit and a sibling mariadb
container. I have imported a dump of the old data as a test. The
dump is ~4gb and imported it takes up about double that; certainly
nothing we need to be too concerned over. The server will be more
than capable of running the db container alongside the lodgeit
instance.
This should have no effect on production until we decide to switch
DNS.
Change-Id: I284864217aa49d664ddc3ebdc800383b2d7e00e3
Point the Zuul scheduler at review01.opendev.org instead of the CNAME
review.opendev.org. This avoids chicken-egg issues because Zuul
actually updates the DNS entries.
Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/798242
Change-Id: I1f6054fdab0fe0fcb311686d6af6454b6a714666
This adds a local mariadb container to the gerrit host to hold the
accountPatchReviewDb database. This is inspired by a few things
- since migration to NoteDB, there is only one table left where
Gerrit records what files have been reviewed for a change. This
logically scales with the number of reviews users are doing.
Pulling the stats on this, we can see since the NoteDB upgrade this
went from a very busy database (~300 queries/70 commits per second)
to barely registering one hit per second :
https://imgur.com/a/QGJV7Fw
Thus separating the db to an external host for performance reasons
is not a large concern any more.
- emperically we've done a bad job in keeping the existing hosted db
up-to-date; it's still running mysql 5.1 and we have been hit by
bugs such as the one referenced in-line which silently drops
backups.
- The other gerrit option is to use an on-disk H2 database. This is
certainly an option, however you need special tools to interact
with it for migration, etc. and it's not safe to backup from files
on disk (as opposed to mysqldump). Upstream advice is unclear, and
varies between H2 being a performance bottleneck to this being
ephemeral data that users don't care about. We know how to admin
mariadb/mysql and this allows us to migrate and backup data, so
seems like the best choice.
- we have a pressing need to update the server to a new operating
system. Running the db alongside the gerrit instance minimises
fiddling we have to do manging connections to and migrating the
hosted db systems.
- related to that, we are tending towards more provider independence
for control-plane servers. A hosted database product is not always
provided, so this gives us more flexibility in moving things
around.
- the main concern here is memory usage. "docker stats" reports a
quiescent container, freshly started on a 8GB host:
gerrit-compose_mariadb_1 67.32MiB
After loading a copy of the production table, and then dumping it
back to a file the same container reports:
gerrit-compose_mariadb_1 462.6MiB
The existing remote mysql configuration path remains mostly the same.
We move the gerrit startup into a script rather than a CMD so we can
call it after a "wait for db" script in the mariadb_container case
(this is the reccommeded way to enforce ordering [1]).
Backups of the local container need different dump commands; backups
are relocated to a new file and updated.
Testing is converted to use this rather than a local H2 database.
[1] https://docs.docker.com/compose/startup-order/
Change-Id: Iec981ef3c2e38889f91e9759e66295dbfb499c2e
This enables the new eavesdrop01.opendev.org server in all current
channels. Puppet has been disabled on the old server and we will
manually stop supybot/meetbot and mirgrate logs before this applies.
Change-Id: I4a422bb9589c8a8761191313a656f8377e93422f
This installs our Limnoira/meetbot container and configures it on
eavesdrop01.opendev.org. I have ported the configuration from the old
puppet as best I can (it is very verbose); my procedure was to use the
Limnoira wizard to start a new config file then backport everything
from the old file. I felt this was best to not miss any new options.
This does channel logging (via built-in ChannelLogger plugin, along
with a cron job for logs2html) and runs our fork of meetbot.
It exports the channel logs via HTTP to /irclogs and meetings logs to
/meetings. meetings.opendev.org will proxy to these two locations
when the server is active.
Note this has not ported the channel list; so the bot will not be
listening in our channels.
Change-Id: I9f9a466c271e1a706f9f98f816de0e84047519f1
This host is no longer under puppet control, but should still be a
webserver to export the logs it is still collecting until we finish
moving that to the new server. Restore the match to open*
See I809f9af3e78f566362142790f6c79654ef5b8959
Change-Id: I524c0a7c5cc93313c180eca68b67a0f0582474df
This moves these services to eavesdrop01.opendev.org, a new
Focal-based server to host IRC services.
We have stopped running puppet on eavesdrop01.openstack.org so there
is nothing left for it to do (note the server is still running
meetbot/ptgbot). Remove the commented out puppet run, and remove the
server from puppet groups. Update the host in the Zuul jobs to the
new node.
Change-Id: I809f9af3e78f566362142790f6c79654ef5b8959
This adds a new server to take over from eavesdrop01.openstack.org.
We limit the puppet installs, etc. to the openstack.org server. The
new server is in the group eavesdrop_opendev as we cut over services.
A stub for basic installation is added to the service playbook.
Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/795004
Change-Id: I88c3059532e4d6ab267fdec5b390daefa5b0c4a1
The openstack-security mailing list is officially closing, and wants
future attempts at posting to end up on openstack-discuss instead:
http://lists.openstack.org/pipermail/openstack-security/2021-June/006077.html
This was also the only remaining user of the notify-impact Gerrit
hook, so we can stop installing/running it.
Change-Id: Id60b781beb072366673b32326e32fd79637c1219
We are trying to replace eavesdrop01.openstack.org
The main landing page serves meeting information which has been moved
to a static site served from AFS at meeting.opendev.org. Redirect
everything to there.
The IRC logs are currently still hosted on eavesdrop01, so while we
work on migrating these, proxy meeting.opendev.org/<irclogs|meetings>
to this server.
Note this will be a no-op until we move the DNS, but we should make
the eavesdrop acme records before merging.
Change-Id: I5c9c23e619dbe930a77f657b5cd6fdd862034301
This site replaces eavesdrop.openstack.org. I think this name makes
more sense.
That is/was being published by jobs directly pushing this onto the
eavesdrop server. Instead, the publishing jobs for irc-meetings now
publish to /afs/openstack.org/project/meetings.opendev.org. This
makes the site available via the static server.
This is actually a production no-op; nothing has changed for the
current publishing. It is still todo to figure out the correct
redirects to keep things working from the existing
eavesdrop.openstack.org and stop the old publishing method.
Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/794085
Change-Id: Ia582c4cee1f074e78cee32626be86fd5eb1d81bd
This cleans up ask-staging which hasn't been a thing in a log time.
We remove some puppet stubs for nodepool builders (they are all ansible
now).
We also cleanup the inventory file to remove corvustest, lists-dev,
pbx, mirror-update*.openstack.org (is opendev.org now), and sort the
LE list.
Change-Id: I8da025640e16bf6e8aca1eb6ec7799d26bd03f12
This will provision LE certs for openstackid.org. If we are happy with
the results then the child change can be merged to to swap apache over
to using the new cert.
Change-Id: Icc9fdd8a39630323916d1f33d9867f93fc6f2b85
We have decided to decommision the ask.openstack.org server as it is
running EOL Xenial, and its manually purchased certiface is about to
expire. Although it has been deprecated for some time, we feel like
it has been around long-enough as a resource that it is best if we
replace it with a place-holder. The links included here are the same
as the currently shown header explaining the site is read-only.
There's nowhere particularly relevant to redirect the site, so we add
a static file here, and some minimal Ansible to put it in the right
place in a generic way in-case we want to do the same for another
service.
Change-Id: I8a31f8fcf9b3064c0ae58e463a6014dc14b518a7
