We are now using the mariadb jdbc connector in production and no longer
need to include the mysql legacy connector in our images. We also don't
need support for h2 or mysql as testing and prod are all using the
mariadb connector and local database.
Note this is a separate change to ensure everything is happy with the
mariadb connector before we remove the fallback mysql connector from our
images.
Change-Id: I982d3c3c026a5351bff567ce7fbb32798718ec1b
Previously we were only managing root's known_hosts via ansible but even
then this wasn't happening because the gerrit_self_hostkey var wasn't
set anywhere. On top of that we need to manage multiple known_hosts
because gerrit must recognize itself and all of the gitea servers.
Update the code to take a dict of host key values and add each entry to
known_hosts for both the root and gerrit2 user.
We remove keyscans from tests to ensure that this update is actually
working.
Change-Id: If64c34322f64c1fb63bf2ebdcc04355fff6ebba2
The extant variable name is never set so this never writes anything
out. Move it to a dictionary value. Use stub values for testing,
this way we don't need the "when:".
Additionally remove an unused old template file.
Change-Id: Id96fde79e28f309aa13e16bdda29f004c3c69c4b
We've stopped relying on jeepyb's track-upstream feature, so stop
installing the entrypoint script and cease running its cronjob.
Depends-On: https://review.opendev.org/799123
Change-Id: I0d6edcc34f25e6bfe2bc41d328ac76618b59f62d
This adds a local mariadb container to the gerrit host to hold the
accountPatchReviewDb database. This is inspired by a few things
- since migration to NoteDB, there is only one table left where
Gerrit records what files have been reviewed for a change. This
logically scales with the number of reviews users are doing.
Pulling the stats on this, we can see since the NoteDB upgrade this
went from a very busy database (~300 queries/70 commits per second)
to barely registering one hit per second :
https://imgur.com/a/QGJV7Fw
Thus separating the db to an external host for performance reasons
is not a large concern any more.
- emperically we've done a bad job in keeping the existing hosted db
up-to-date; it's still running mysql 5.1 and we have been hit by
bugs such as the one referenced in-line which silently drops
backups.
- The other gerrit option is to use an on-disk H2 database. This is
certainly an option, however you need special tools to interact
with it for migration, etc. and it's not safe to backup from files
on disk (as opposed to mysqldump). Upstream advice is unclear, and
varies between H2 being a performance bottleneck to this being
ephemeral data that users don't care about. We know how to admin
mariadb/mysql and this allows us to migrate and backup data, so
seems like the best choice.
- we have a pressing need to update the server to a new operating
system. Running the db alongside the gerrit instance minimises
fiddling we have to do manging connections to and migrating the
hosted db systems.
- related to that, we are tending towards more provider independence
for control-plane servers. A hosted database product is not always
provided, so this gives us more flexibility in moving things
around.
- the main concern here is memory usage. "docker stats" reports a
quiescent container, freshly started on a 8GB host:
gerrit-compose_mariadb_1 67.32MiB
After loading a copy of the production table, and then dumping it
back to a file the same container reports:
gerrit-compose_mariadb_1 462.6MiB
The existing remote mysql configuration path remains mostly the same.
We move the gerrit startup into a script rather than a CMD so we can
call it after a "wait for db" script in the mariadb_container case
(this is the reccommeded way to enforce ordering [1]).
Backups of the local container need different dump commands; backups
are relocated to a new file and updated.
Testing is converted to use this rather than a local H2 database.
[1] https://docs.docker.com/compose/startup-order/
Change-Id: Iec981ef3c2e38889f91e9759e66295dbfb499c2e
The openstack-security mailing list is officially closing, and wants
future attempts at posting to end up on openstack-discuss instead:
http://lists.openstack.org/pipermail/openstack-security/2021-June/006077.html
This was also the only remaining user of the notify-impact Gerrit
hook, so we can stop installing/running it.
Change-Id: Id60b781beb072366673b32326e32fd79637c1219
This doesn't install of Focal; moving forward we'll either use H2 or a
container database. Just remove this package for now.
Change-Id: I69cdcdddc1ba0e0cf4ef5f8ba705bcd3a2afa689
Due to [1] --all-databases is no longer working with our version of
database. Move to explicitly backing up the only database we care
about now, which is accountPatchReviewDb; everything else is in
notedb.
[1] https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1914695
Change-Id: Iab2a8ab612cc0a0f10c90123f2936c0abda9e76f
Add facility to borg-backup role to run a command and save the output
of it to a separate archive file during the backup process.
This is mostly useful for database backups. Compressed on-disk logs
are terrible for differential backups because revisions have
essentially no common data. By saving the uncompressed stream
directly from mysqldump, we allow borg the chance to de-duplicate,
saving considerable space on the backup servers.
This is implemented for our ansible-managed servers currently doing
dumps. We also add it to the testinfra.
This also separates the archive names for the filesystem and stream
backup with unique prefixes so they can be pruned separately.
Otherwise we end up keeping only one of the stream or filesystem
backups which isn't the intention. However, due to issues with
--append-only mode we are not issuing prune commands at this time.
Note the updated dump commands are updated slightly, particularly with
"--skip-extended-insert" which was suggested by mordred and
significantly improves incremental diff-ability by being slightly more
verbose but keeping much more of the output stable across dumps.
Change-Id: I500062c1c52c74a567621df9aaa716de804ffae7
By setting the auth type to DEVELOPMENT_BECOME_ANY_ACCOUNT and passing
--dev to the init process, gerrit will create an initial admin user
for us. We leverage this user to create a sample project, change,
Zuul user and sample CI result comment.
We also update testinfra to take some screenshots of gerrit and report
them back.
Change-Id: I56cda99790d3c172e10b664e57abeca10efc5566
We were setting these values in gerrit.config but it isn't clear if
these now need to go in jgit.config instead. I've tried to clarify with
upstream maintainers as the documentation is quite confusing. While we
wait for clarification why don't we just set the values in both files to
ensure we are covered.
This converts jgit.config to a jinja2 template so that we can use the
variable number of pack files setting.
Change-Id: I70c1e6b738ed6e9fdb72d86e7cf3fb8cfecf1323
The receive.autogc configuration apparently needs to be in its own
file, not in the general gerrit.conf. Move it to the correct
location.
https://review.opendev.org/Documentation/config-gerrit.html#jgit-receive
While we're here, correct the filename on the gerrit.config and
secure.config templates to make it clear they're jinja2 files, and
add a file mode to the replication.config where it was missing.
Change-Id: I9243bccac103c51ee435725aae482731642a37cc
The gate testing sets gerrit_database_type to 'h2', but the container
does not setup /var/gerrit/db. It ends up owned by root in the
container and gerrit can't write to it, causing traceback errors for
account_patch_reviews when anything is updated.
Add a db peer directory with the other config dirs, and make sure it
is bind mounted into the image in the same way.
While we are here, alphabetise this list of directories for
consistency.
Change-Id: Ib24e3027622a711db76d24172b27cdebe6152cb6
Currently is is 0700 and owned by root, so the gerrit2 user can't
access the keys in there by default (in production this is already
correct). This is the key used to push replication to gitea for
example.
Change-Id: Ie0230cd74aeb3e759a597c999b2507260f2b6c17
When we get to Gerrit 3.x the old html theming, hideci.js and
commentlinks that parse html and rely on urls no longer work. Lets clean
that up when we get there.
We can add back in similar things that polygerrit does support later
but we don't want that to make the upgrade even more difficult to do.
This should only be merged once we are running Gerrit 3.x.
Change-Id: I838840e6cbf09ca28faeb2cf06290e298a4a1f74
This is a followup to the previous change that cleans up a now unused
crontab resource. THis should be merged once the actual crontab entry
has been removed from the host.
Change-Id: Ieb8df262d9b047e860a22cfac5569c4ca0b7111b
We stopped serving this content and the next step is to stop managing it
internally. This depends on a change to jeepyb that makes the local git
dir management on the jeepyb side optional. Once that lands we can
update our configs to tell jeepyb to stop managing it.
We also stop doing garbage collection, mounting it into containers that
don't need it, etc.
Depends-On: https://review.opendev.org/758597
Change-Id: I2185e90edfcac71941bc29a4e11b7b2d4c7c2e13
The title.svg logo for opendev and two jquery js files are no longer
managed by ansible nor do they appear to be in our docker image. They
appear to have been lost when we converted from puppet to ansible +
docker. Add them back in. We are also missing the icla html content (but
not the other clas) add this one back in.
We vendor the js contents even though in the past we copied them from a
git repo clone and a distro package installation. This way we don't have
unexpected surprises, record that the files are used, and can always
update them later.
Change-Id: I981b4b0f233ece45d03a80dc1724a4e496f66eb8
The host is review-test.opendev.org, so hostvars for
review-test.openstack.org are not so much going to do anything.
It's easier if we just ssh as root from review to gerrit2
on review-test.
review-test needs to be in letsencrypt group and have a
handler.
We need to install mysql - it's on the existing review
servers but not in ansible, it's just left over from
puppet.
The db credentials are in /root/.gerrit_db.cnf
Change-Id: I90e3c9d1b398cc16fea9f7056cfb059c7140160e
Private host_vars exist on bridge separate from review01.openstack.org
with their own mysql database, rest key and email key. Add in a
conditional on accepting our own hostkey since we're not actually
going to run manage-projects on this host so we don't need to do that.
Change-Id: If7f45250e7ead62f2117f3e6c0fcfc0b4e7f71aa
We want to use stop_grace_period to manage gerrit service stops. This
feature was added in docker-compose 1.10 but the distro provides 1.5.
Work around this by installing docker-compose from pypi.
This seems like a useful feature and we want to manage docker-compose
the same way globally so move docker-compose installation into the
install-docker role.
New docker-compose has slightly different output that we must check for
in the gitea start/stop machinery. We also need to check for different
container name formatting in our test cases. We should pause here and
consider if this has any upgrade implications for our existing services.
Change-Id: Ia8249a2b84a2ef167ee4ffd66d7a7e7cff8e21fb
We use project-config for gerrit, gitea and nodepool config. That's
cool, because can clone that from zuul too and make sure that each
prod run we're doing runs with the contents of the patch in question.
Introduce a flag file that can be touched in /home/zuulcd that will
block zuul from running prod playbooks. By default, if the file is
there, zuul will wait for an hour before giving up.
Rename zuulcd to zuul
To better align prod and test, name the zuul user zuul.
Change-Id: I83c38c9c430218059579f3763e02d6b9f40c7b89
We use this in some utlity scripts, but we currently don't
write it out anywhere. It was an old puppet artifact.
Change-Id: Ib6fdfc4f4a9c5d1befdb6d256989450996dd2a3d
We run some utility scripts which ssh to ourselves, but we aren't
setting host keys for them. We should fix that.
Change-Id: I2aa5d5e65b15c5c151767377dbc5ead1e442b3ce
Files are bind-mounted into the container in different locations.
Set envvars pointing to the right places.
Also - we need to bind-mount the projects.yaml and projects.ini
files into the container.
While we're at it, move patchset-created to be a regular file.
Change-Id: Iacd3e921464b24479db13bbf7ae998b8d8e2103d
These still exist, because puppet put them there. We should delete
them by hand on the server and then land this.
Change-Id: I7ee0c60fade0b29a52898a7edf64554d1aa61cae
When we run manage-projects, we always want the latest image.
Add a task file to the gerrit role so that it can have access
to role default variables. Then call it from the playbook
with tasks_from.
Change-Id: I08bd10539d7f08e8759f1c22d66dea18821c9d42
jeepyb is installed in the gerrit image because of hook scripts.
To run manage-projects and track-upstream, make wrapper scripts
in /usr/local that runs the commands from in the container image
bind-mounting the appropriate dirs and files into the container.
Change-Id: I7ef1f00e69d4c310d69d83c80ca210e8f340878d
This is in anticipation of running manage-projects, which wants to
use the dev subdir of project-config when run on dev, but we're
currently not doing anything with that. Point at root or dev as
appropriate. Then we can do a similar thing - probably just
bind-mounting the root or the dev when we docker run for
manage-projects.
Change-Id: Ia13bbb0bf8dbe1f7e7c0f378ba9b41bef3ecd5c1
We install jeepyb and launchpadlib in gerrit-base. Those are
important. We also need to add cgi for gitweb.
The gerrit init command does two things that we don't actually
want it to do at runtime. It extracts the plugins into the
plugins dir, and it downloads the right database library.
We can extract the plugins for it during image creation, and
then we can also download the plugin it would have downloaded.
We can also download the mysql library for it:
https://gerrit.googlesource.com/gerrit/+/refs/heads/stable-2.13/gerrit-pgm/src/main/resources/com/google/gerrit/pgm/init/libraries.config
Finally, we tell it to not download or expand anything during
init, because we're running in a container and next time we run
the process that dir isn't going to be there.
Our gerrit integration tests don't depend on our gerrit image builds.
Put in image depends between run-review and gerrit builds.
We also need to depend directly on opendev-buildset-registry.
Add java.security.egd setting to java invocation
This tells java to be secure.
https://stackoverflow.com/questions/58991966/what-java-security-egd-option-is-for
Add support for setting heap limit properly
The gerrit init script does this based on the value in
container.javaOptions. We could, but then we'd have to
run an entrypoint script. Instead, set the value via
the JAVA_OPTIONS env var setting based on a value from
ansible.
Finally, make gerrit-master image build non-voting
It looks like there might be a real issue, but debugging that
is not important for us at this moment.
Depends-On: https://review.opendev.org/714216
Change-Id: I01e94c10f470fb3c8ddfce7b0e201357e5050679
We have a need to run ad-hoc commands with the various things
mounted, such as the gerrit init, as well as our friends like
manage-projects. Make a second "service" that doesn't do anything,
but allows us to run "docker-compose run".
Change-Id: Ic9bafe85b90af413c891b969273437be9df6a79f
We don't want to copy static and etc into the static and etc dirs
creating static/static and etc/etc.
It's possible this could also be done with trailing /'s?
Change-Id: I054158a677261743ed67107894e207f7b615ea7f
For our rollout, we need to be able to run this without actually
running the up.
Also, split out startup tasks so that we can run them from a
dedicated start playbook by themselves.
Change-Id: I08d994e496fbd8d5adbfa1ce344b0ae52f46535c
We rolled out review-dev with podman and it worked fine for us. It
worked less fine for nodepool-builder, although we still might be
able to solve it. Maybe right now isn't the time to do this switch.
Gitea, gitea-lb and zuul-registry all use docker instead of podman.
The only thing running with podman right now is review-dev. We can
do a manual cleanup of podman there before runnign this to keep
things simple:
- stop gerrit service
- uninstall podman and podman-compose
- uninstall podman ppa config
- uninstall pip3
Then let ansible install docker and docker compose up.
Story: #2007407
Task: #39062
Change-Id: I9bf99b18559d49d11ba99a96f02a4a45a4f65a86
While we're in there - fix a misspelling.
Remove auth.restTokenPrivateKey from config file. It hasn't been
used since 2.6: https://gerrit-review.googlesource.com/c/gerrit/+/70770
Change-Id: I94405cf870d57780b86f30c2bddb573ff15c05bc
Make image and volume list in compose file templated.
Rename the gerrit-podman directory to not be based on tool.
Make sure we run the job on changes to the playbooks.
Update the job name - it's not just review-dev anymore.
Change-Id: I0341fa95caff656a2176cc2026ec0ac8903fb24e
