1520 Commits

Author SHA1 Message Date
Zuul
b412c96f9e Merge "Clean up cron tab entry from ansible once removed from host" 2020-11-23 06:42:08 +00:00
Zuul
e48ac000e3 Merge "codesearch: Add robots.txt" 2020-11-23 05:41:33 +00:00
Zuul
d755c64080 Merge "Remove reviewdb config from Gerrit" 2020-11-22 22:58:04 +00:00
Zuul
fc09bc3668 Merge "Clean up old Gerrit html theming and commentlinks" 2020-11-22 20:20:43 +00:00
Clark Boylan
1ccf5e68e5 Remove reviewdb config from Gerrit
This should only land once we are on Gerrit 3.x and happy with it. But
at this point the mysql reviewdb will not be used anymore and config for
it can be removed. We keep general mysql things like tools and backups
in place as the accountPatchReviewDb continues to live in MySQL.

This also comments out calls to jeepyb's welcome-message,
update-blueprint and update-bug entrypoints from the patchset-created
event hook, since they rely on database connections for the moment.
Calls to update-bug in change-abandoned and change-merged event
hooks are retained as those code paths don't rely on database
interaction nor attempt to load the removed configuration.

Change-Id: I6e24dbb223fd3f76954db3dd74a03887cf2e2a8b
2020-11-22 17:53:49 +00:00
Clark Boylan
01dab90bc1 Handle x/ prefix projects on gerrit 3.2
Gerrit seems to handle x/ for plugin extensions in polygerrit.
Unfortunately we've got projects called x/* and that breaks cloning of
these projects. Lets just avoid that for nwo until we can do a rename.

Change-Id: Id01739725c22af9d02ac30b1653743b49a35a332
2020-11-21 13:58:34 -08:00
Jeremy Stanley
f3e6cade06 Disable private changes in Gerrit
More recent Gerrit versions have replaced teh old drafts feature
with a combination of private changes and work in progress state.
The latter might be useful eventually and could be used to augment
or replace our workflow -1 votes, but the not-so-private nature of
drafts is what caused us to disable them and we should do the same
with private changes as they'll become an attractive nuisance
otherwise.

Change-Id: I213a73b0ba6a3dd2a8ce402d6a396e6c494529c8
2020-11-21 18:30:27 +00:00
James E. Blair
4a0964066e Add X-Forwarded-Proto to gerrit apache config
This fixes directly linked changes redirecting to an incorrect port.
Example: https://review.opendev.org/712697 =>
  Location: https://review.opendev.org:80/c/openstack/nova/+/712697/

https://bugs.chromium.org/p/gerrit/issues/detail?id=13701

Change-Id: I750c9048b85f119b309ab676b930a201d81f9099
2020-11-21 10:09:35 -08:00
fungi.admin
2197f11a0f Merge "Omnibus Gerrit 3.2 changes" 2020-11-21 17:19:58 +00:00
Zuul
03edbd8b14 Merge "docker: install rsyslog to capture container output" 2020-11-20 09:12:23 +00:00
Ian Wienand
1288de67aa codesearch: Add robots.txt
We don't want anything on the codesearch page indexed

Change-Id: I556b77013cf1b7ff2c03426fea92a6d445131f6d
2020-11-20 19:13:32 +11:00
Zuul
e07a739a29 Merge "base: Remove rsyslogd reinstall" 2020-11-20 03:35:53 +00:00
Ian Wienand
fc82ee2c5b codesearch: redirect other hostnames to codesearch.opendev.org
Make sure things like https://codesearch01.opendev.org or
https://codesearch.openstack.org all get redirected to the main
http://codesearch.opendev.org site.

Change-Id: Ifb2c810425b60474ba3581611d85ddcc5c77f130
2020-11-20 11:48:00 +11:00
Zuul
1b16dae681 Merge "Migrate codesearch site to container" 2020-11-19 22:26:12 +00:00
Ian Wienand
343c626c5e base: Remove rsyslogd reinstall
This reverts the changes made with
e0fc90cd067647ffcf06e0bfb84fe11636d33be5 as it has been deployed.

Change-Id: If5de429d2259a151c5e4c22fab0c6588341465e1
2020-11-20 07:41:43 +11:00
Ian Wienand
368466730c Migrate codesearch site to container
The hound project has undergone a small re-birth and moved to

 https://github.com/hound-search/hound

which has broken our deployment.  We've talked about leaving
codesearch up to gitea, but it's not quite there yet.  There seems to
be no point working on the puppet now.

This builds a container than runs houndd.  It's an opendev specific
container; the config is pulled from project-config directly.

There's some custom scripts that drive things.  Some points for
reviewers:

 - update-hound-config.sh uses "create-hound-config" (which is in
   jeepyb for historical reasons) to generate the config file.  It
   grabs the latest projects.yaml from project-config and exits with a
   return code to indicate if things changed.

 - when the container starts, it runs update-hound-config.sh to
   populate the initial config.  There is a testing environment flag
   and small config so it doesn't have to clone the entire opendev for
   functional testing.

 - it runs under supervisord so we can restart the daemon when
   projects are updated.  Unlike earlier versions that didn't start
   listening till indexing was done, this version now puts up a "Hound
   is not ready yet" message when while it is working; so we can drop
   all the magic we were doing to probe if hound is listening via
   netstat and making Apache redirect to a status page.

 - resync-hound.sh is run from an external cron job daily, and does
   this update and restart check.  Since it only reloads if changes
   are made, this should be relatively rare anyway.

 - There is a PR to monitor the config file
   (https://github.com/hound-search/hound/pull/357) which would mean
   the restart is unnecessary.  This would be good in the near and we
   could remove the cron job.

 - playbooks/roles/codesearch is unexciting and deploys the container,
   certificates and an apache proxy back to localhost:6080 where hound
   is listening.

I've combined removal of the old puppet bits here as the "-codesearch"
namespace was already being used.

Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
2020-11-20 07:41:12 +11:00
Ian Wienand
3cd8cd0765 devel job: use ansible-core name
As described inline, installing ansible from source now installs the
"ansible-core" package, instead of "ansible-base".  Since they can't
live together nicely, we have to do a manual override for the devel
job.

Change-Id: I1299ea330e6de048b661fc087f016491758631c7
2020-11-18 14:49:46 +11:00
Clark Boylan
52094c441d Clean up old Gerrit html theming and commentlinks
When we get to Gerrit 3.x the old html theming, hideci.js and
commentlinks that parse html and rely on urls no longer work. Lets clean
that up when we get there.

We can add back in similar things that polygerrit does support later
but we don't want that to make the upgrade even more difficult to do.

This should only be merged once we are running Gerrit 3.x.

Change-Id: I838840e6cbf09ca28faeb2cf06290e298a4a1f74
2020-11-17 16:04:58 -08:00
Clark Boylan
57f9e54ad8 Omnibus Gerrit 3.2 changes
These changes are squashed together to simplify applying them to config
management without zuul and ansible running one of these without the
others. We essentially need them all in place at the same time to
accurately reflect the post upgrade state.

We stop blocking /p/ in gerrit's apache vhost. /p/ is used for
dashboards.

We add a few java options that new gerrit sets by default.

We update the gerrit image in docker compose to 3.2.

We update zuul to use basic auth instead of digest auth when talking to
Gerrit.

Change-Id: I6ea38313544ce1ecbc4cfd914b1f33e77d0d2d03
2020-11-17 16:04:56 -08:00
Zuul
c6a835ecc4 Merge "Stop managing gerrit's local git mirror dir" 2020-11-17 22:13:24 +00:00
Zuul
2c7591c318 Merge "Set gerrit.serverId in gerrit.config" 2020-11-17 21:22:53 +00:00
Zuul
0221406ac1 Merge "Disable change.move and enableSignedPush in gerrit" 2020-11-17 21:22:50 +00:00
Zuul
947572bda9 Merge "Add example maintenance message for Gerrit server" 2020-11-17 20:36:30 +00:00
Javier Pena
70aff57869 Switch CentOS 7 mirror source to kernel.org mirror
The current source for our sync is outdated [1], and does not include
the latest CentOS 7.9 content. This is breaking some jobs relying on
CentOS, since they may have more recent packages in the image than
in the repositories [2].

[1] - https://mirror-status.centos.org/
[2] - https://bugs.launchpad.net/tripleo/+bug/1904214

Change-Id: Ie404396b0f108121241bdc26dd8d22c85106b0c4
2020-11-16 11:34:07 +01:00
Zuul
fb072cd1b0 Merge "Remove rsyslogd xconsole workaround" 2020-11-16 06:13:32 +00:00
Jeremy Stanley
0f3ef01027 Add example maintenance message for Gerrit server
Include comments in the Gerrit vhost config template to make setting
a temporary site-wide maintenance message easier.

Change-Id: I81f69185e081b2a6506d5355bb07a90cb3e03fea
2020-11-13 20:00:53 +00:00
Ian Wienand
e2ab4a3f4b borg-backup: excludes updates
No need to backup /etc/project-config or root cache things

Change-Id: If31844e974b0bb287c871721453bc6ad500604a8
2020-11-12 15:25:34 +11:00
Ian Wienand
93b4c9ed1f borg-backup: space out cron jobs evenly
To avoid the backup jobs running over the top of each other, space the
cron jobs out evenly through the day for each server.

Change-Id: I07a096ee130e61e9efc89887d627da8ef829854a
2020-11-11 12:57:32 +11:00
Clark Boylan
b9b1cba959 Set gerrit.serverId in gerrit.config
This serverId value is used by notedb to identify the gerrit cluster
that notedb contents belong to. By default a random uuid is generated by
gerrit for this value. In order to avoid config management and gerrit
fighting over this value after we upgrade we set a value now.

This should be safe to land on 2.13 as old gerrit should ignore the
value.

Change-Id: I57c9b436a9d0d1dfe77eee907d50fc1dcda6ab12
2020-11-10 10:30:58 -08:00
Zuul
d3b275b32c Merge "borg-backup: ignore .bup files" 2020-11-10 02:09:59 +00:00
Ian Wienand
4c243338e3 borg-backup: ignore .bup files
We don't need to backup the old backup tracking files

Change-Id: I829a0f29c016618156e8dca7387d22bb7f0d9d60
2020-11-10 12:10:59 +11:00
Zuul
de7899b5ff Merge "mirror-update/reprepro : use common functions" 2020-11-10 00:59:37 +00:00
Zuul
71dd0b65df Merge "etherpad: reduce backup rotations" 2020-11-10 00:10:55 +00:00
Zuul
d11949817d Merge "Add all backup hosts to borg backups" 2020-11-09 23:39:51 +00:00
Ian Wienand
1006f478c1 etherpad: reduce backup rotations
The database dumps are ~3gb each, so are starting to fill up the disk.
Reduce on-disk rotations to 2 and rely on remote backups more.

Change-Id: Ie8fc8fd7c4c36ca37e6813104f4c711ea55186ab
2020-11-10 09:24:34 +11:00
Zuul
d3a53e8ec0 Merge "Remove mirror-update server and related puppet" 2020-11-09 21:07:11 +00:00
Ian Wienand
d533e89089 Add all backup hosts to borg backups
Backups have been going well on ethercalc02, so add borg backup runs
to all backed-up servers.  Port in some additional excludes for Zuul
and slightly modify the /var/ matching.

Change-Id: Ic3adfd162fa9bedd84402e3c25b5c1bebb21f3cb
2020-11-09 17:23:22 +11:00
Zuul
375ef4c005 Merge "grafana: redirect http to CNAME" 2020-11-05 22:31:17 +00:00
Ian Wienand
99ee86bce5 grafana: redirect http to CNAME
Currently the http site redirects to https://grafana01.opendev.org
which is a bit confusing.  Change this to redirect to the CNAME
grafana.opendev.org.

Change-Id: I174ce06af036a8f74ace6b4d95edaaf85f63fa27
2020-11-05 13:27:09 +11:00
Ian Wienand
eb07ab3613 borg-backup: add fuse
Add the FUSE dependencies for our hosts backed up with borg, along
with a small script to make mounting the backups easier.  This is the
best way to recover something quickly in what is sure to be a
stressful situation.

Documentation and testing is updated.

Change-Id: I1f409b2df952281deedff2ff8f09e3132a2aff08
2020-11-05 11:56:46 +11:00
Zuul
1bc5ceba98 Merge "borg-backup: disambiguate for multiple servers" 2020-11-04 04:15:36 +00:00
Ian Wienand
d9d9a53cb7 borg-backup: disambiguate for multiple servers
The ssh config and cron job will overwrite each other when we have
multiple backup servers.

Ensure the markers are different.

Change-Id: I1736fa9c72c90a357b2229bc86c33b33a2bb321c
2020-11-04 13:11:43 +11:00
Ian Wienand
bdc56b1caf letsencrypt-install-txt-record: pause after adding TXT records
The rdnc reload should notify the child nameservers which will update
quickly, but for general sanity pause 1 minute to allow for
propagation of the added authentication records before continuing.

Change-Id: Ic0f9398e056df77c96824eff8215395947997d82
2020-10-30 16:49:31 +11:00
Zuul
b72845c274 Merge "Cleanup grafana.openstack.org" 2020-10-29 05:15:00 +00:00
Ian Wienand
166b009613 reprepro: fix apt-puppetlabs volume name
This was missing the leading "apt-"

Change-Id: I61c9823972cf4333232a606c37f2349b0c6c7831
2020-10-29 14:09:42 +11:00
Ian Wienand
b22abc02ff reprepro: catch stderr of individual deb-docker runs
This should catch the stderr of each individual run into the separate
logfile.

Change-Id: Iace3511ee8ec876debfc7361ff814b1d181e1458
2020-10-29 09:41:19 +11:00
Ian Wienand
3ddbba92ae reprepro: fix cron config path and randomise times better
The config should use the full path to the config directory, append
/etc/reprepro to the job.

Currently all the reprepro jobs hash to the same start time because it
uses the hostname as a seed.  Use the unique string name as the seed
so each job starts at a unique time.

Change-Id: If2745d0cd274f390dbff6337b7a44093b5919908
2020-10-29 09:27:29 +11:00
Zuul
3d3f1def31 Merge "reprepo: enable cron jobs" 2020-10-28 21:45:42 +00:00
Ian Wienand
c49ece9204 Cleanup grafana.openstack.org
The opendev.org server is in production, cleanup the old puppet-based
host.

Change-Id: I6db3ce929226a23b96234b52ece8b17f4c6a326a
2020-10-29 07:59:42 +11:00
Zuul
8cc54626d0 Merge "nameserver: Allow master server to notify via ipv6" 2020-10-28 11:26:23 +00:00