This doesn't install of Focal; moving forward we'll either use H2 or a
container database. Just remove this package for now.
Change-Id: I69cdcdddc1ba0e0cf4ef5f8ba705bcd3a2afa689
Now that the update-bug script no longer tries to update bug
assignments, it's possible to run it on patchset-created events
again. Go back to doing that until someone has time to build a
suitable replacement for it.
This partially reverts commit
1ccf5e68e51815479381a941fd9cf4f469498c6d.
Change-Id: Idf589eb818d208d65d1f1430ddec962b015165c0
Depends-On: https://review.opendev.org/782538
Due to [1] --all-databases is no longer working with our version of
database. Move to explicitly backing up the only database we care
about now, which is accountPatchReviewDb; everything else is in
notedb.
[1] https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1914695
Change-Id: Iab2a8ab612cc0a0f10c90123f2936c0abda9e76f
Add facility to borg-backup role to run a command and save the output
of it to a separate archive file during the backup process.
This is mostly useful for database backups. Compressed on-disk logs
are terrible for differential backups because revisions have
essentially no common data. By saving the uncompressed stream
directly from mysqldump, we allow borg the chance to de-duplicate,
saving considerable space on the backup servers.
This is implemented for our ansible-managed servers currently doing
dumps. We also add it to the testinfra.
This also separates the archive names for the filesystem and stream
backup with unique prefixes so they can be pruned separately.
Otherwise we end up keeping only one of the stream or filesystem
backups which isn't the intention. However, due to issues with
--append-only mode we are not issuing prune commands at this time.
Note the updated dump commands are updated slightly, particularly with
"--skip-extended-insert" which was suggested by mordred and
significantly improves incremental diff-ability by being slightly more
verbose but keeping much more of the output stable across dumps.
Change-Id: I500062c1c52c74a567621df9aaa716de804ffae7
By setting the auth type to DEVELOPMENT_BECOME_ANY_ACCOUNT and passing
--dev to the init process, gerrit will create an initial admin user
for us. We leverage this user to create a sample project, change,
Zuul user and sample CI result comment.
We also update testinfra to take some screenshots of gerrit and report
them back.
Change-Id: I56cda99790d3c172e10b664e57abeca10efc5566
This is enabled on review-test if you want to test it out. It is
intended to speed up fetches and clones and such as you don't have to
list and interact with all of the gerrit change refs to do those common
operations.
Change-Id: I65b430548a2805cd05dc4cdbcf9354a9c18faadc
We are seeing java gc go crazy at times and aren't quite sure what is
causing it. Add jvm gc logging to the gerrit process to help us identify
what is happening.
Additionally we add SYS_PTRACE to the container capabilities so that you
can get heap dumps from the jvm. To get a heap dump you need to do
roughyl:
docker exec -u root -it gerritcontainerid jhsdb jmap --heap --pid $pid
Change-Id: Ib4a5b84fda4eca73c7971c31ee74c3232eb733e4
We were setting these values in gerrit.config but it isn't clear if
these now need to go in jgit.config instead. I've tried to clarify with
upstream maintainers as the documentation is quite confusing. While we
wait for clarification why don't we just set the values in both files to
ensure we are covered.
This converts jgit.config to a jinja2 template so that we can use the
variable number of pack files setting.
Change-Id: I70c1e6b738ed6e9fdb72d86e7cf3fb8cfecf1323
Gerrit 3.2 supports java 11 now and Gerrit 3.3 will be the last to
support java 8. Lets get ahead of things and switch to java 11.
Change-Id: I1b2f6b1bdadad10917ef5c56ce77f7d7cfc8625d
The receive.autogc configuration apparently needs to be in its own
file, not in the general gerrit.conf. Move it to the correct
location.
https://review.opendev.org/Documentation/config-gerrit.html#jgit-receive
While we're here, correct the filename on the gerrit.config and
secure.config templates to make it clear they're jinja2 files, and
add a file mode to the replication.config where it was missing.
Change-Id: I9243bccac103c51ee435725aae482731642a37cc
For unfathomable reasons, Gerrit implements automatic GC on every
push and enables this by default but recommends in the documentation
that it be turned off. Follow their recommendation on this, since it
seems to result in additional load and we already periodically GC
all repos anyway.
Change-Id: I9a46c69b26e0a746f2aed308a28e5408e5c34ef1
We're seeing high system load and decreased performance on our
production Gerrit instance. Some research suggests this may be I/O
contention which can be relieved through better caching:
https://groups.google.com/g/repo-discuss/c/7CemrH4lVJE
According to `gerrit show-caches --show-jvm --show-threads` some of
our memory-only caches are already at their default maximums after
only a few days of operation, and one in particular
(changeid_project) has a particularly poor cache hit ratio of 24% at
the moment. Increase changeid_project from the 1024 entry default by
32x (manual tests at 8x approached 50% cache hit), increase projects
by 4x (greater than the number of repos we host for now), and double
the others (groups_bysubgroup, permission_sort) since they still had
reasonable cache hit ratios while full. Also alpha-order the
existing cache overrides in our config for improved maintainability.
This will require a Gerrit service restart to take effect, once the
file update has been deployed.
Change-Id: Ieecd1802ce53cc0d37c68476b94b44cbe36fbd6e
The gate testing sets gerrit_database_type to 'h2', but the container
does not setup /var/gerrit/db. It ends up owned by root in the
container and gerrit can't write to it, causing traceback errors for
account_patch_reviews when anything is updated.
Add a db peer directory with the other config dirs, and make sure it
is bind mounted into the image in the same way.
While we are here, alphabetise this list of directories for
consistency.
Change-Id: Ib24e3027622a711db76d24172b27cdebe6152cb6
Currently is is 0700 and owned by root, so the gerrit2 user can't
access the keys in there by default (in production this is already
correct). This is the key used to push replication to gitea for
example.
Change-Id: Ie0230cd74aeb3e759a597c999b2507260f2b6c17
We're seeing a growing backlog of email events that all must funnel
through the single thread allocated to sending email. We think this may
be related to server slowness that we are observing. Bump the number of
threads to 4 to see if that flushes the queues quicker and gives us a
more responsive server.
Change-Id: I730c8f657191cedb46f81f4abc0e1796ef847b27
This should only land once we are on Gerrit 3.x and happy with it. But
at this point the mysql reviewdb will not be used anymore and config for
it can be removed. We keep general mysql things like tools and backups
in place as the accountPatchReviewDb continues to live in MySQL.
This also comments out calls to jeepyb's welcome-message,
update-blueprint and update-bug entrypoints from the patchset-created
event hook, since they rely on database connections for the moment.
Calls to update-bug in change-abandoned and change-merged event
hooks are retained as those code paths don't rely on database
interaction nor attempt to load the removed configuration.
Change-Id: I6e24dbb223fd3f76954db3dd74a03887cf2e2a8b
More recent Gerrit versions have replaced teh old drafts feature
with a combination of private changes and work in progress state.
The latter might be useful eventually and could be used to augment
or replace our workflow -1 votes, but the not-so-private nature of
drafts is what caused us to disable them and we should do the same
with private changes as they'll become an attractive nuisance
otherwise.
Change-Id: I213a73b0ba6a3dd2a8ce402d6a396e6c494529c8
When we get to Gerrit 3.x the old html theming, hideci.js and
commentlinks that parse html and rely on urls no longer work. Lets clean
that up when we get there.
We can add back in similar things that polygerrit does support later
but we don't want that to make the upgrade even more difficult to do.
This should only be merged once we are running Gerrit 3.x.
Change-Id: I838840e6cbf09ca28faeb2cf06290e298a4a1f74
These changes are squashed together to simplify applying them to config
management without zuul and ansible running one of these without the
others. We essentially need them all in place at the same time to
accurately reflect the post upgrade state.
We stop blocking /p/ in gerrit's apache vhost. /p/ is used for
dashboards.
We add a few java options that new gerrit sets by default.
We update the gerrit image in docker compose to 3.2.
We update zuul to use basic auth instead of digest auth when talking to
Gerrit.
Change-Id: I6ea38313544ce1ecbc4cfd914b1f33e77d0d2d03
Include comments in the Gerrit vhost config template to make setting
a temporary site-wide maintenance message easier.
Change-Id: I81f69185e081b2a6506d5355bb07a90cb3e03fea
This serverId value is used by notedb to identify the gerrit cluster
that notedb contents belong to. By default a random uuid is generated by
gerrit for this value. In order to avoid config management and gerrit
fighting over this value after we upgrade we set a value now.
This should be safe to land on 2.13 as old gerrit should ignore the
value.
Change-Id: I57c9b436a9d0d1dfe77eee907d50fc1dcda6ab12
This is a followup to the previous change that cleans up a now unused
crontab resource. THis should be merged once the actual crontab entry
has been removed from the host.
Change-Id: Ieb8df262d9b047e860a22cfac5569c4ca0b7111b
We stopped serving this content and the next step is to stop managing it
internally. This depends on a change to jeepyb that makes the local git
dir management on the jeepyb side optional. Once that lands we can
update our configs to tell jeepyb to stop managing it.
We also stop doing garbage collection, mounting it into containers that
don't need it, etc.
Depends-On: https://review.opendev.org/758597
Change-Id: I2185e90edfcac71941bc29a4e11b7b2d4c7c2e13
change.move is a new option in gerrit 3.0 that toggles whether or not
the change move api is enabled. We disable it because there are
potential confusing side effect for moving a change with parent changes
then merging the moved change. Details can be found in
https://bugs.chromium.org/p/gerrit/issues/detail?id=9877
We've not needed to move chagnes previously and users can abandon and
push to a different branch instead.
With enableSignedPush we set that to false even though it is an existing
default because newer gerrit seems to write it out to its config file.
We write it out to avoid unnecessary file updates after the upgrade.
Note I believe it is safe to land this on 2.13 or 2.16 as gerrit should
just ignore change.move until we get to 3.x and enableSignedPush already
defaults to false.
Change-Id: I9db2026b1e5cafefd448f33f74d6b7b60efafdb4
The title.svg logo for opendev and two jquery js files are no longer
managed by ansible nor do they appear to be in our docker image. They
appear to have been lost when we converted from puppet to ansible +
docker. Add them back in. We are also missing the icla html content (but
not the other clas) add this one back in.
We vendor the js contents even though in the past we copied them from a
git repo clone and a distro package installation. This way we don't have
unexpected surprises, record that the files are used, and can always
update them later.
Change-Id: I981b4b0f233ece45d03a80dc1724a4e496f66eb8
We remove old git web server env vars from the apache config and add
comments to our /p/ handling to describe the need for further cleanup
when Gerrit is upgraded.
Change-Id: I79fc130dec0a8b00706c0ec0f8fcab4d867e34d1
Gerrit is repurposing the /p/ path for project dashboard under
polygerrit. We use this path for Git mirrors. To resolve this let's
disable the /p/ path now then when it is used for project dashboards
users won't be as confused.
This has the added benefit of reducing the number of mirrors we need to
manage which makes managing branches in the mirrors simpler.
Change-Id: I9ebca2049a4a0707ecfbaecd92e42ebc1e6c3f87
The host is review-test.opendev.org, so hostvars for
review-test.openstack.org are not so much going to do anything.
It's easier if we just ssh as root from review to gerrit2
on review-test.
review-test needs to be in letsencrypt group and have a
handler.
We need to install mysql - it's on the existing review
servers but not in ansible, it's just left over from
puppet.
The db credentials are in /root/.gerrit_db.cnf
Change-Id: I90e3c9d1b398cc16fea9f7056cfb059c7140160e
Private host_vars exist on bridge separate from review01.openstack.org
with their own mysql database, rest key and email key. Add in a
conditional on accepting our own hostkey since we're not actually
going to run manage-projects on this host so we don't need to do that.
Change-Id: If7f45250e7ead62f2117f3e6c0fcfc0b4e7f71aa
The ssl flag is deprecated and we get cronspam [0] warning us about
this. The docs [1] say we should use ssl-mode instead.
[0] WARNING: --ssl is deprecated and will be removed in a future version. Use --ssl-mode instead.
[1] https://dev.mysql.com/doc/refman/5.7/en/connection-options.html#option_general_ssl
Change-Id: I060bbfeaf1171dac50dcfcd2c62fcaa8956fb4e2
By default gerrit replication pushes +refs/*:refs/*, which includes
refs/changes. For large repositories that potentially means hundreds
of thousands of references.
Per-repo git mirroring does not push refs/changes, so when it runs it
ends up deleting those references, which can take a long time, blocking
the executor.
To fix that, we should:
- stop pushing refs/changes to GitHub (this change)
- delete refs/changes on GitHub repositories, asynchronously
- enable per-repo replication
- disable Gerrit-wide replication
NB: it is unclear if Gerrit replication would start deleting the
extraneous references on remote GitHub repositories once this
merges. If this is the case, since replication is limited to a
single thread (default value for 'threads') and is not happening
in an executor, this should not have negative impact, beyond
potentially delaying GitHub mirroring.
Change-Id: I94f69c889c9b4418ef81b3b2ca436ba99696ba72
Also add a 5 minute stop grace period. This lines up with the init
script we were using when this was a systemd managed service.
Change-Id: I5a92bb214b96447008ad570e176adda13c4ca0cb
We want to use stop_grace_period to manage gerrit service stops. This
feature was added in docker-compose 1.10 but the distro provides 1.5.
Work around this by installing docker-compose from pypi.
This seems like a useful feature and we want to manage docker-compose
the same way globally so move docker-compose installation into the
install-docker role.
New docker-compose has slightly different output that we must check for
in the gitea start/stop machinery. We also need to check for different
container name formatting in our test cases. We should pause here and
consider if this has any upgrade implications for our existing services.
Change-Id: Ia8249a2b84a2ef167ee4ffd66d7a7e7cff8e21fb
