This installs statusbot on eavesdrop01.opendev.org.
Otherwise it's just config translation and bringing up the daemon.
Change-Id: I246b2723372594e65bcd1ba90215d6831d4c0c72
This moves these services to eavesdrop01.opendev.org, a new
Focal-based server to host IRC services.
We have stopped running puppet on eavesdrop01.openstack.org so there
is nothing left for it to do (note the server is still running
meetbot/ptgbot). Remove the commented out puppet run, and remove the
server from puppet groups. Update the host in the Zuul jobs to the
new node.
Change-Id: I809f9af3e78f566362142790f6c79654ef5b8959
It looks like something changed in a header field that makes this not
match. Use a less specific string that should.
Change-Id: I55b2c63e026bf2b68821a9a0302a82d5728ab114
We are trying to replace eavesdrop01.openstack.org
The main landing page serves meeting information which has been moved
to a static site served from AFS at meeting.opendev.org. Redirect
everything to there.
The IRC logs are currently still hosted on eavesdrop01, so while we
work on migrating these, proxy meeting.opendev.org/<irclogs|meetings>
to this server.
Note this will be a no-op until we move the DNS, but we should make
the eavesdrop acme records before merging.
Change-Id: I5c9c23e619dbe930a77f657b5cd6fdd862034301
This site replaces eavesdrop.openstack.org. I think this name makes
more sense.
That is/was being published by jobs directly pushing this onto the
eavesdrop server. Instead, the publishing jobs for irc-meetings now
publish to /afs/openstack.org/project/meetings.opendev.org. This
makes the site available via the static server.
This is actually a production no-op; nothing has changed for the
current publishing. It is still todo to figure out the correct
redirects to keep things working from the existing
eavesdrop.openstack.org and stop the old publishing method.
Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/794085
Change-Id: Ia582c4cee1f074e78cee32626be86fd5eb1d81bd
ARA's master branch now has static site generation, so we can move
away from the stable branch and get the new reports.
In the mean time ARA upstream has moved to github, so this updates the
references for the -devel job.
Depends-On: https://review.opendev.org/c/openstack/project-config/+/793530
Change-Id: I008b35562994f1205a4f66e53f93b9885a6b8754
We have decided to decommision the ask.openstack.org server as it is
running EOL Xenial, and its manually purchased certiface is about to
expire. Although it has been deprecated for some time, we feel like
it has been around long-enough as a resource that it is best if we
replace it with a place-holder. The links included here are the same
as the currently shown header explaining the site is read-only.
There's nowhere particularly relevant to redirect the site, so we add
a static file here, and some minimal Ansible to put it in the right
place in a generic way in-case we want to do the same for another
service.
Change-Id: I8a31f8fcf9b3064c0ae58e463a6014dc14b518a7
This zuul02 instance will replace zuul01. There are a few items to
coordinate when doing an actual switch so we haven't removed zuul01 from
inventory here. In particular we need to update gearman server config
values in the zuul cluster and we need to save queues, shutdown zuul01,
then start zuul02's scheduler and restore queues there.
I believe landing this change is safe as we don't appear to start zuul
on new instances by default. Reviewers should double check this.
Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/791039
Change-Id: I524b456e494124d8293fbe8e1468de40f3800772
This converts our existing puppeted mailman configuration into a set of
ansible roles and a new playbook. We don't try to do anything new and
instead do our best to map from puppet to ansible as closely as
possible. This helps reduce churn and will help us find problems more
quickly if they happen.
Followups will further cleanup the puppetry.
Change-Id: If8cdb1164c9000438d1977d8965a92ca8eebe4df
We will be rotating zk01-03.openstack.org out and replacing them with
zk04-06.opendev.org. This is the first change in that process which puts
zk04 into the rotation. This should only be landed when operators are
ready to manually stop zookeeper on zk03 (which is being replaced by
zk04 in this change).
Change-Id: Iea69130f6b3b2c8e54e3938c60e4a3295601c46f
This handles planet.openstack.org and redirects it to the
opendev.org/openstack/planet-openstack repo, where we will put a
README and the OPML file of the last state as we deprecate this
service.
Change-Id: If141aca5efbdbe60c91ceefaa4e05c98cd0ba5bb
This adds a program, zookeeper-statsd, which monitors zookeeper
metrics and reports them to statsd. It also adds a container to
run that program. And it runs the container on each of the
ZooKeeper quorum members. And it updates the graphite host to
allow statsd traffic from quorum members. And it updates the
4-letter-word whitelist to allow the mntr command (which is used
to gather metrics) to be issued.
Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
This adds the new focal nodepool launchers replacements for nl02-04 to
our inventory. This will configure them with an idle configuration. We
then confirm they are happy running in an idle state then switch over
the config from the old to new servers.
Depends-On: https://review.opendev.org/c/openstack/project-config/+/780982
Change-Id: Iea645925caaeee6f498aa690c4f2c848f6899317
Zookeeper supports a number of "4 letter" commands [0] which are useful
for debugging and general diagnostics. By default only srvr is enabled,
but we want to add stat and dump to see details on server and client
connection statuses.
We do this via the 4lw.commands.whitelist configuration option [1] and
not the docker image env vars because we're mounting a zoo.cfg in
already.
[0] https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_4lw
[1] https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_clusterOptions
Change-Id: I24ea9b37cd5766c9d393106e8eab34623cad1624
This is a new focal replacement for nl01.openstack.org. We keep
nl01.openstack.org in our inventory for now because we want ansible to
update the nodepool.yaml configs for these two hosts to coordinate a
hand off of responsibilities once we are happy with the new deployment.
We also switch the testing hostname to nl04.openstack.org as this will
be the last nodepool launcher to be removed. When we swap it out the
testing will be updated to use focal hosts.
Depends-On: https://review.opendev.org/c/openstack/project-config/+/779863
Change-Id: Ib3ea6586fe0567c1edf6255ee9be50164d35db62
The previous refstack server had 'api' in the endpoint
addresses of API calls. Let's try to set it in the new
instance as well to keep the same interface.
Also, fix the typo in the testinfra host match and in
the test name.
Change-Id: I7319990144396b3a753678975a09b0add3ac4465
This checks the backup archives and alerts us if anything seems wrong.
This will take a few hours, so we run once a week.
Change-Id: I832c0d29a37df94d4bf2704c59bb3f8d855c3cc8
This adds a dockerfile to build an opendevorg/refstack image as well as
the jobs to build and publish it.
Change-Id: Icade6c713fa9bf6ab508fd4d8d65debada2ddb30
Previously the test was checking that stderr reported "Cloning into
$PATH" which also happens in failure cases. We add an explicit check for
a successful command return code to ensure that we aren't failing with
that output.
Change-Id: Iec51217f2cc97e6a56ff9d8b7a260650010f229f
This will help us catch any regressions in our workaround for handling
x/ routes in Gerrit. We update the test project in Gerrit to be
x/test-project then add a testinfra test to clone it.
If we can clone x/test-project then our workaround continues to
function.
Change-Id: I50e4cb1a5d3c9f7c4405500f09bf6c3be3e7df9c
This reworks the gerrit testing slightly to give some broader
coverage.
It sets up ssh keys for the user; not really necessary but can be
helpful when interacting on a held host.
It sets up groups and verification labels just so Zuul can comment
with -2/+2; again this is not really necessary, but makes things a
little closer to production reality.
We make multiple changes, so we can better test navigating between
them. The change comments are updated to have some randomness in them
so they don't all look the same. We take screen shots of two change
pages to validate the navigation between them.
Change-Id: I60b869e4fdcf8849de836e33db643743128f8a70
This installs the zuul-summary-results plugin into our gerrit
container. testinfra is updated to take a screenshot of the plugin in
action.
Change-Id: Ie0a165cc6ffc765c03457691901a1dd41ce99d5a
By setting the auth type to DEVELOPMENT_BECOME_ANY_ACCOUNT and passing
--dev to the init process, gerrit will create an initial admin user
for us. We leverage this user to create a sample project, change,
Zuul user and sample CI result comment.
We also update testinfra to take some screenshots of gerrit and report
them back.
Change-Id: I56cda99790d3c172e10b664e57abeca10efc5566
This server is canonicallly named review01.openstack.org in inventory.
We need to use that inventory name in our testing.
Change-Id: I1d16469f5abb764978945b5209e01a4e7d2ccb3d
We don't need to test two servers in this test; remove review-dev.
Consensus seems to be this was for testing plans that have now been
superseded.
Change-Id: Ia4db5e0748e1c82838000c9b655808c3d8b74461
To complete our transition to borg backups, remove bup-related bits
from backup hosts. All hosts have been backing up with borg since
Ic3adfd162fa9bedd84402e3c25b5c1bebb21f3cb.
Change-Id: Ie99f8cee9befee28bcf74bff9f9994c4b17b87ff
The hound project has undergone a small re-birth and moved to
https://github.com/hound-search/hound
which has broken our deployment. We've talked about leaving
codesearch up to gitea, but it's not quite there yet. There seems to
be no point working on the puppet now.
This builds a container than runs houndd. It's an opendev specific
container; the config is pulled from project-config directly.
There's some custom scripts that drive things. Some points for
reviewers:
- update-hound-config.sh uses "create-hound-config" (which is in
jeepyb for historical reasons) to generate the config file. It
grabs the latest projects.yaml from project-config and exits with a
return code to indicate if things changed.
- when the container starts, it runs update-hound-config.sh to
populate the initial config. There is a testing environment flag
and small config so it doesn't have to clone the entire opendev for
functional testing.
- it runs under supervisord so we can restart the daemon when
projects are updated. Unlike earlier versions that didn't start
listening till indexing was done, this version now puts up a "Hound
is not ready yet" message when while it is working; so we can drop
all the magic we were doing to probe if hound is listening via
netstat and making Apache redirect to a status page.
- resync-hound.sh is run from an external cron job daily, and does
this update and restart check. Since it only reloads if changes
are made, this should be relatively rare anyway.
- There is a PR to monitor the config file
(https://github.com/hound-search/hound/pull/357) which would mean
the restart is unnecessary. This would be good in the near and we
could remove the cron job.
- playbooks/roles/codesearch is unexciting and deploys the container,
certificates and an apache proxy back to localhost:6080 where hound
is listening.
I've combined removal of the old puppet bits here as the "-codesearch"
namespace was already being used.
Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
Add the FUSE dependencies for our hosts backed up with borg, along
with a small script to make mounting the backups easier. This is the
best way to recover something quickly in what is sure to be a
stressful situation.
Documentation and testing is updated.
Change-Id: I1f409b2df952281deedff2ff8f09e3132a2aff08
In converting this to ansible I forgot to install the reprepro keytab.
The encoded secret has been added for production.
Change-Id: I39d586e375ad96136cc151a7aed6f4cd5365f3c7
This started with me wondering why gerritbot was putting all its
output into /var/log/syslog -- it turns out Xenial docker is
configured to use journalctl (which forwards to syslog) and Bionic
onwards uses json-file.
Both are sub-optimial; but particularly the json-file because we lose
the logs when the container dies. This proposes moving to a more
standard model of having the containers log to syslog and redirecting
that to files on disk.
Install a rsyslog configuration to capture "docker-*" program names
and put them into logfiles in /var/log/containers. Also install
rotation for these files.
In an initial group of docker-compose files, setup logging to syslog
which should then be captured into these files. Add some basic
testing.
If this works OK, I think we can standardise our docker-compose files
like this to caputure the logs the same everywhere.
Change-Id: I940a5b05057e832e2efad79d9a2ed5325020ed0c
