We are currently re-chowning the running db directories back to root,
causing havoc for the db. Drop the explicit permissions to avoid
this.
Change-Id: I8d8ce5c62c660875d5c6eed54c686996576ec9df
This removes the old config to choose the old change screen by default
as everything is polygerrit now.
We remove the pre plugin melody config as melody now ships as a plugin
and has separate configuration.
We remove old theming information as that is supplied via external files
now.
We remove anonymous git download config because we don't set
gerrit.canonicalGitUrl which is required for this to work. We don't set
that because we don't have a git:// server anymore.
Bump the lucene thread count from 4 to 8 as we have more cores on the
system we run on.
Finally add some comments to help make sense of config that is left in
place.
Change-Id: Ie0b48e544191839067e66647d2ea32f74ce19ed3
The gerrit config diff after the 3.3 ugprade [1] seems to remove some
quotes. We also quote the bug URL, because it seems to think the
trailing # is a comment now.
[1] https://etherpad.opendev.org/p/gerrit-upgrade-3.3
Change-Id: I3ca0ec925a0e6da33a1cbe2333c118b1baa7257c
This bumps the gerrit image up to our 3.3 image. Followup changes will
shift upgrade testing to test 3.3 to 3.4 upgrades, clean up no longer
needed 3.2 images, and start building 3.4 images.
Change-Id: Id0f544846946d4c50737a54ceb909a0a686a594e
This file used to be managed by puppet-gerrit and it seems we missed
converting it to Ansible. Add it with the contents from the server.
Change-Id: I10a10166446941d2676ae9181fc74b5a1408c5ed
Copy static files directly into the container image instead of
managing them dynamically with Ansible.
Change-Id: I0ebe40ad2a97e87b00137af7c93a3ffa84929a2e
We are seeing that replication tasks occasionally sit around forever and
have had to take manual intervention. One theory is that this is related
to networking between the gerrit server and the gitea servers. We don't
set maxRetries which means replication should be retried infinitely
which means if we hit the timeout we should try again. 15 minutes was
sort of arbitrarily chosen as ~twice the time it takes to clone a large
repo like nova.
Change-Id: Iec2536ad149a2e625a1f0107b9fcee3079493607
We are now using the mariadb jdbc connector in production and no longer
need to include the mysql legacy connector in our images. We also don't
need support for h2 or mysql as testing and prod are all using the
mariadb connector and local database.
Note this is a separate change to ensure everything is happy with the
mariadb connector before we remove the fallback mysql connector from our
images.
Change-Id: I982d3c3c026a5351bff567ce7fbb32798718ec1b
Upstream stable-3.2 and stable-3.3 branches have been fixed to allow us
to use the mariadb jdbc connector. The previous change has updated our
images to ensure they include this fix. We can now update the config to
use the mariadb connector.
Change-Id: I43ac20d601ff88e42f0d20387fc6ad8842ab8244
Previously we were only managing root's known_hosts via ansible but even
then this wasn't happening because the gerrit_self_hostkey var wasn't
set anywhere. On top of that we need to manage multiple known_hosts
because gerrit must recognize itself and all of the gitea servers.
Update the code to take a dict of host key values and add each entry to
known_hosts for both the root and gerrit2 user.
We remove keyscans from tests to ensure that this update is actually
working.
Change-Id: If64c34322f64c1fb63bf2ebdcc04355fff6ebba2
After I replace the docker packages some services I thought would come
back did not.
Lodegit seems to be an oversight, add restart always.
Also make sure the ZK containers start themselves.
I believe with Gerrit we've made the choice to not start automatically
due to the general high-touch nature of restarts. Keep the database
consistent and remove the auto restart there.
Change-Id: I98fa3055ac269564ed96570df0700b2aad24e4d2
The extant variable name is never set so this never writes anything
out. Move it to a dictionary value. Use stub values for testing,
this way we don't need the "when:".
Additionally remove an unused old template file.
Change-Id: Id96fde79e28f309aa13e16bdda29f004c3c69c4b
This moves review02 out of the review-staging group and into the main
review group. At this point, review01.openstack.org is inactive so we
can remove all references to openstack.org from the groups. We update
the system-config job to run against a focal production server, and
remove the unneeded rsync setup used to move data.
This additionally enables replication; this should be a no-op when
applied as part of the transition process is to manually apply this,
so that DNS setup can pull zone changes from opendev.org.
It also switches to the mysql connector, as noted inline we found some
issues with mariadb.
Note backups follow in a separate step to avoid doing too much at
once, hence dropping the backup group from the testing list.
Change-Id: I7ee3e3051ea8f3237fd5f6bf1dcc3e5996c16d10
The services: tag was accidentally put inside the mariadb section with
Iec981ef3c2e38889f91e9759e66295dbfb499c2e. This works in the gate
because it uses this path, but fails on current production. Move it
outside.
Change-Id: I8b6009da6271f451f123831a16801a9f0bd5374f
We've stopped relying on jeepyb's track-upstream feature, so stop
installing the entrypoint script and cease running its cronjob.
Depends-On: https://review.opendev.org/799123
Change-Id: I0d6edcc34f25e6bfe2bc41d328ac76618b59f62d
This adds a local mariadb container to the gerrit host to hold the
accountPatchReviewDb database. This is inspired by a few things
- since migration to NoteDB, there is only one table left where
Gerrit records what files have been reviewed for a change. This
logically scales with the number of reviews users are doing.
Pulling the stats on this, we can see since the NoteDB upgrade this
went from a very busy database (~300 queries/70 commits per second)
to barely registering one hit per second :
https://imgur.com/a/QGJV7Fw
Thus separating the db to an external host for performance reasons
is not a large concern any more.
- emperically we've done a bad job in keeping the existing hosted db
up-to-date; it's still running mysql 5.1 and we have been hit by
bugs such as the one referenced in-line which silently drops
backups.
- The other gerrit option is to use an on-disk H2 database. This is
certainly an option, however you need special tools to interact
with it for migration, etc. and it's not safe to backup from files
on disk (as opposed to mysqldump). Upstream advice is unclear, and
varies between H2 being a performance bottleneck to this being
ephemeral data that users don't care about. We know how to admin
mariadb/mysql and this allows us to migrate and backup data, so
seems like the best choice.
- we have a pressing need to update the server to a new operating
system. Running the db alongside the gerrit instance minimises
fiddling we have to do manging connections to and migrating the
hosted db systems.
- related to that, we are tending towards more provider independence
for control-plane servers. A hosted database product is not always
provided, so this gives us more flexibility in moving things
around.
- the main concern here is memory usage. "docker stats" reports a
quiescent container, freshly started on a 8GB host:
gerrit-compose_mariadb_1 67.32MiB
After loading a copy of the production table, and then dumping it
back to a file the same container reports:
gerrit-compose_mariadb_1 462.6MiB
The existing remote mysql configuration path remains mostly the same.
We move the gerrit startup into a script rather than a CMD so we can
call it after a "wait for db" script in the mariadb_container case
(this is the reccommeded way to enforce ordering [1]).
Backups of the local container need different dump commands; backups
are relocated to a new file and updated.
Testing is converted to use this rather than a local H2 database.
[1] https://docs.docker.com/compose/startup-order/
Change-Id: Iec981ef3c2e38889f91e9759e66295dbfb499c2e
The update_blueprint script has been updated to call the Gerrit REST API
instead of relying on the old Gerrit DB.
Depends-On: https://review.opendev.org/c/opendev/jeepyb/+/795912
Change-Id: Ie21ee33801429ef4398f70b22223ee1e9bea1301
The openstack-security mailing list is officially closing, and wants
future attempts at posting to end up on openstack-discuss instead:
http://lists.openstack.org/pipermail/openstack-security/2021-June/006077.html
This was also the only remaining user of the notify-impact Gerrit
hook, so we can stop installing/running it.
Change-Id: Id60b781beb072366673b32326e32fd79637c1219
With a pure javascript plugin, dropping a new file in the plugins/
directory and reloading the page is sufficient to see changes.
However, with .jar plugins (as zuul-summary-plugin now is) you need to
actually issue a reload, which requires the included permissions.
Enable it dev mode, which is where you'll very likely be trying to
iterate development with a change to a plugin. I don't think it's
really that dangerous for production, but traditionally it's been off
there so let's leave it like that.
While we're here, write out a little script to help you quickly deploy
a new .jar of the plugin when we're testing.
Change-Id: I57fa18755f8a8168da12c48f1f38d272da1c6599
This doesn't install of Focal; moving forward we'll either use H2 or a
container database. Just remove this package for now.
Change-Id: I69cdcdddc1ba0e0cf4ef5f8ba705bcd3a2afa689
Now that the update-bug script no longer tries to update bug
assignments, it's possible to run it on patchset-created events
again. Go back to doing that until someone has time to build a
suitable replacement for it.
This partially reverts commit
1ccf5e68e51815479381a941fd9cf4f469498c6d.
Change-Id: Idf589eb818d208d65d1f1430ddec962b015165c0
Depends-On: https://review.opendev.org/782538
Due to [1] --all-databases is no longer working with our version of
database. Move to explicitly backing up the only database we care
about now, which is accountPatchReviewDb; everything else is in
notedb.
[1] https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1914695
Change-Id: Iab2a8ab612cc0a0f10c90123f2936c0abda9e76f
Add facility to borg-backup role to run a command and save the output
of it to a separate archive file during the backup process.
This is mostly useful for database backups. Compressed on-disk logs
are terrible for differential backups because revisions have
essentially no common data. By saving the uncompressed stream
directly from mysqldump, we allow borg the chance to de-duplicate,
saving considerable space on the backup servers.
This is implemented for our ansible-managed servers currently doing
dumps. We also add it to the testinfra.
This also separates the archive names for the filesystem and stream
backup with unique prefixes so they can be pruned separately.
Otherwise we end up keeping only one of the stream or filesystem
backups which isn't the intention. However, due to issues with
--append-only mode we are not issuing prune commands at this time.
Note the updated dump commands are updated slightly, particularly with
"--skip-extended-insert" which was suggested by mordred and
significantly improves incremental diff-ability by being slightly more
verbose but keeping much more of the output stable across dumps.
Change-Id: I500062c1c52c74a567621df9aaa716de804ffae7
By setting the auth type to DEVELOPMENT_BECOME_ANY_ACCOUNT and passing
--dev to the init process, gerrit will create an initial admin user
for us. We leverage this user to create a sample project, change,
Zuul user and sample CI result comment.
We also update testinfra to take some screenshots of gerrit and report
them back.
Change-Id: I56cda99790d3c172e10b664e57abeca10efc5566
This is enabled on review-test if you want to test it out. It is
intended to speed up fetches and clones and such as you don't have to
list and interact with all of the gerrit change refs to do those common
operations.
Change-Id: I65b430548a2805cd05dc4cdbcf9354a9c18faadc
We are seeing java gc go crazy at times and aren't quite sure what is
causing it. Add jvm gc logging to the gerrit process to help us identify
what is happening.
Additionally we add SYS_PTRACE to the container capabilities so that you
can get heap dumps from the jvm. To get a heap dump you need to do
roughyl:
docker exec -u root -it gerritcontainerid jhsdb jmap --heap --pid $pid
Change-Id: Ib4a5b84fda4eca73c7971c31ee74c3232eb733e4
We were setting these values in gerrit.config but it isn't clear if
these now need to go in jgit.config instead. I've tried to clarify with
upstream maintainers as the documentation is quite confusing. While we
wait for clarification why don't we just set the values in both files to
ensure we are covered.
This converts jgit.config to a jinja2 template so that we can use the
variable number of pack files setting.
Change-Id: I70c1e6b738ed6e9fdb72d86e7cf3fb8cfecf1323
Gerrit 3.2 supports java 11 now and Gerrit 3.3 will be the last to
support java 8. Lets get ahead of things and switch to java 11.
Change-Id: I1b2f6b1bdadad10917ef5c56ce77f7d7cfc8625d
The receive.autogc configuration apparently needs to be in its own
file, not in the general gerrit.conf. Move it to the correct
location.
https://review.opendev.org/Documentation/config-gerrit.html#jgit-receive
While we're here, correct the filename on the gerrit.config and
secure.config templates to make it clear they're jinja2 files, and
add a file mode to the replication.config where it was missing.
Change-Id: I9243bccac103c51ee435725aae482731642a37cc
For unfathomable reasons, Gerrit implements automatic GC on every
push and enables this by default but recommends in the documentation
that it be turned off. Follow their recommendation on this, since it
seems to result in additional load and we already periodically GC
all repos anyway.
Change-Id: I9a46c69b26e0a746f2aed308a28e5408e5c34ef1
We're seeing high system load and decreased performance on our
production Gerrit instance. Some research suggests this may be I/O
contention which can be relieved through better caching:
https://groups.google.com/g/repo-discuss/c/7CemrH4lVJE
According to `gerrit show-caches --show-jvm --show-threads` some of
our memory-only caches are already at their default maximums after
only a few days of operation, and one in particular
(changeid_project) has a particularly poor cache hit ratio of 24% at
the moment. Increase changeid_project from the 1024 entry default by
32x (manual tests at 8x approached 50% cache hit), increase projects
by 4x (greater than the number of repos we host for now), and double
the others (groups_bysubgroup, permission_sort) since they still had
reasonable cache hit ratios while full. Also alpha-order the
existing cache overrides in our config for improved maintainability.
This will require a Gerrit service restart to take effect, once the
file update has been deployed.
Change-Id: Ieecd1802ce53cc0d37c68476b94b44cbe36fbd6e
The gate testing sets gerrit_database_type to 'h2', but the container
does not setup /var/gerrit/db. It ends up owned by root in the
container and gerrit can't write to it, causing traceback errors for
account_patch_reviews when anything is updated.
Add a db peer directory with the other config dirs, and make sure it
is bind mounted into the image in the same way.
While we are here, alphabetise this list of directories for
consistency.
Change-Id: Ib24e3027622a711db76d24172b27cdebe6152cb6
Currently is is 0700 and owned by root, so the gerrit2 user can't
access the keys in there by default (in production this is already
correct). This is the key used to push replication to gitea for
example.
Change-Id: Ie0230cd74aeb3e759a597c999b2507260f2b6c17
We're seeing a growing backlog of email events that all must funnel
through the single thread allocated to sending email. We think this may
be related to server slowness that we are observing. Bump the number of
threads to 4 to see if that flushes the queues quicker and gives us a
more responsive server.
Change-Id: I730c8f657191cedb46f81f4abc0e1796ef847b27
This should only land once we are on Gerrit 3.x and happy with it. But
at this point the mysql reviewdb will not be used anymore and config for
it can be removed. We keep general mysql things like tools and backups
in place as the accountPatchReviewDb continues to live in MySQL.
This also comments out calls to jeepyb's welcome-message,
update-blueprint and update-bug entrypoints from the patchset-created
event hook, since they rely on database connections for the moment.
Calls to update-bug in change-abandoned and change-merged event
hooks are retained as those code paths don't rely on database
interaction nor attempt to load the removed configuration.
Change-Id: I6e24dbb223fd3f76954db3dd74a03887cf2e2a8b
More recent Gerrit versions have replaced teh old drafts feature
with a combination of private changes and work in progress state.
The latter might be useful eventually and could be used to augment
or replace our workflow -1 votes, but the not-so-private nature of
drafts is what caused us to disable them and we should do the same
with private changes as they'll become an attractive nuisance
otherwise.
Change-Id: I213a73b0ba6a3dd2a8ce402d6a396e6c494529c8
When we get to Gerrit 3.x the old html theming, hideci.js and
commentlinks that parse html and rely on urls no longer work. Lets clean
that up when we get there.
We can add back in similar things that polygerrit does support later
but we don't want that to make the upgrade even more difficult to do.
This should only be merged once we are running Gerrit 3.x.
Change-Id: I838840e6cbf09ca28faeb2cf06290e298a4a1f74
