All hosts are now running thier backups via borg to servers in
vexxhost and rax.ord.
For reference, the servers being backed up at this time are:
borg-ask01
borg-ethercalc02
borg-etherpad01
borg-gitea01
borg-lists
borg-review-dev01
borg-review01
borg-storyboard01
borg-translate01
borg-wiki-update-test
borg-zuul01
This removes the old bup backup hosts, the no-longer used ansible
roles for the bup backup server and client roles, and any remaining
bup related configuration.
For simplicity, we will remove any remaining bup cron jobs on the
above servers manually after this merges.
Change-Id: I32554ca857a81ae8a250ce082421a7ede460ea3c
The hound project has undergone a small re-birth and moved to
https://github.com/hound-search/hound
which has broken our deployment. We've talked about leaving
codesearch up to gitea, but it's not quite there yet. There seems to
be no point working on the puppet now.
This builds a container than runs houndd. It's an opendev specific
container; the config is pulled from project-config directly.
There's some custom scripts that drive things. Some points for
reviewers:
- update-hound-config.sh uses "create-hound-config" (which is in
jeepyb for historical reasons) to generate the config file. It
grabs the latest projects.yaml from project-config and exits with a
return code to indicate if things changed.
- when the container starts, it runs update-hound-config.sh to
populate the initial config. There is a testing environment flag
and small config so it doesn't have to clone the entire opendev for
functional testing.
- it runs under supervisord so we can restart the daemon when
projects are updated. Unlike earlier versions that didn't start
listening till indexing was done, this version now puts up a "Hound
is not ready yet" message when while it is working; so we can drop
all the magic we were doing to probe if hound is listening via
netstat and making Apache redirect to a status page.
- resync-hound.sh is run from an external cron job daily, and does
this update and restart check. Since it only reloads if changes
are made, this should be relatively rare anyway.
- There is a PR to monitor the config file
(https://github.com/hound-search/hound/pull/357) which would mean
the restart is unnecessary. This would be good in the near and we
could remove the cron job.
- playbooks/roles/codesearch is unexciting and deploys the container,
certificates and an apache proxy back to localhost:6080 where hound
is listening.
I've combined removal of the old puppet bits here as the "-codesearch"
namespace was already being used.
Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
This script has been moved into management done by ansible and is
executing on mirror-update not afsdb01. Cleanup the unused dead code.
Change-Id: Idc1c10cc968eef5ec1aeece70bad7606a7607269
Previous review pointed out some additional modules we probably
aren't using any longer.
Remove the openafs::client section from openstack_project::server
because we're doing this with ansible now.
Depends-On: https://review.opendev.org/733890
Change-Id: Ib5104da9cf7d53b77191f48ec185f5d667d51944
This autogenerates the list of ssl domains for the ssl-cert-check tool
directly from the letsencrypt list.
The first step is the install-certcheck role that replaces the
puppet-ssl_cert_check module that does the same. The reason for this
is so that during gate testing we can test this on the test
bridge.openstack.org server, and avoid adding another node as a
requirement for this test.
letsencrypt-request-certs is updated to set a fact
letsencrypt_certcheck_domains for each host that is generating a
certificate. As described in the comments, this defaults to the first
host specified for the certificate and the listening port can be
indicated (if set, this new port value is stripped when generating
certs as is not necessary for certificate generation).
The new letsencrypt-config-certcheck role runs and iterates all
letsencrypt hosts to build the final list of domains that should be
checked. This is then extended with the
letsencrypt_certcheck_additional_domains value that covers any hosts
using certificates not provisioned by letsencrypt using this
mechanism.
These additional domains are pre-populated from the openstack.org
domains in the extant check file, minus those openstack.org domain
certificates we are generating via letsencrypt (see
letsencrypt-create-certs/handlers/main.yaml). Additionally, we
update some of the certificate variables in host_vars that are
listening on port !443.
As mentioned, bridge.openstack.org is placed in the new certcheck
group for gate testing, so the tool and config file will be deployed
to it. For production, cacti is added to the group, which is where
the tool currently runs. The extant puppet installation is disabled,
pending removal in a follow-on change.
Change-Id: Idbe084f13f3684021e8efd9ac69b63fe31484606
Remove the separate "mirror_opendev" group and rename it to just
"mirror". Update various parts to reflect that change.
We no longer deploy any mirror hosts with puppet, remove the various
configuration files.
Depends-On: https://review.opendev.org/728345
Change-Id: Ia982fe9cb4357447989664f033df976b528aaf84
Zuul is publishing lovely container images, so we should
go ahead and start using them.
We can't use containers for zuul-executor because of the
docker->bubblewrap->AFS issue, so install from pip there.
Don't start any of the containers by default, which should
let us safely roll this out and then do a rolling restart.
For things (like web or mergers) where it's safe to do so,
a followup change will swap the flag.
Change-Id: I37dcce3a67477ad3b2c36f2fd3657af18bc25c40
Extract eavedrop into its own service playbook and
puppet manifest. While doing that, stop using jenkinsuser
on eavesdrop in favor of zuul-user.
Add the ability to override the keys for the zuul user.
Remove openstack_project::server, it doesn't do anything.
Containerize and anisblize accessbot. The structure of
how we're doing it in puppet makes it hard to actually
run the puppet in the gate. Run the script in its own
playbook so that we can avoid running it in the gate.
Change-Id: I53cb63ffa4ae50575d4fa37b24323ad13ec1bac3
Make a service playbook, manifest and jobs for codesearch.
Remove openstack_project::server - it doesn't do anything.
Change-Id: I44c140de4ae0b283940f8e23e8c47af983934471
We have one global variable that is used in two places.
By removing it, we can more easily split site.pp into
per-service manifest files, and ultimately we should be
deriving this from groups['elasticsearch'] anyway.
Change-Id: I1d794b269847da85778f71e816359953af9b31e0
Migration plan:
* add zk* to emergency
* copy data files on each node to a safe place for DR backup
* make a json data backup: zk-shell localhost:2181 --run-once 'mirror / json://!tmp!zookeeper-backup.json/'
* manually run a modified playbook to set up the docker infra without starting containers
* rolling restart; for each node:
* stop zk
* split data and log files and move them to new locations
* remove zk packages
* start zk containers
* remove from emergency; land this change.
Change-Id: Ic06c9cf9604402aa8eb4bb79238021c14c5d9563
This should be mostly a no-op - but we will need to do a shutdown
in emergency mode.
Tell the gerrit role to not run compose up when run as part of
remote_puppet_git.
Change-Id: Id45376c2697656a12afeacf317b6f26c85c08dad
We previously had two manually issued certs (one each for opendev.org
and openstack.org) but now have a single cert with all the appropriate
names in it automatically issued by LE. Use this new cert before the old
one expires.
Change-Id: I635d2bfd820fe138ee951833dd66f157b2b7c097
Removed all variables related to Silverstripe
Dependency
Change-Id: Ib5e6834686c4952dd8e7220a31abe71a9278e397
Signed-off-by: smarcet <smarcet@gmail.com>
wiki, status, and single node ci should all run on xenial now. Switch
their testing to xenial from trusty.
Change-Id: I3a0c2faa47f2ec17809e3845c7226173188def63
The review-dev service playbook should do everything now that
the puppet did. Update how we're running things.
Change-Id: I70303c48328ea6713c24bf9c6f63d4808d30b95c
This provisions the cert but does not use it yet. We will do the
switchover once the cert is confirmed to be in place.
Depends-On: https://review.opendev.org/701819
Change-Id: I04fee48b9a79758527d8f9e8128c0fa915cd133e
We were setting the cert file contents to the paths rather than updating
the paths to point at the new LE certs. Fix this by setting the _file
vars which update the path.
This includes a partial revert of the previous change to not switch
git.zuul-ci.org over to LE as we haven't provisioned an LE cert for it
yet.
Change-Id: I41c2aa1d03afba4ebf6378e9abf8276154666df7
This switches the zuul-ci.org/zuulci.org vhost to use newly issued
letsencrypt certs. It also does the same for git.zuul-ci.org, which
is a different vhost. Since that vhost is tied into a configuration
which can't accept cert file paths (only content), adjust it to use
the newer "website" manifest pattern which can.
Change-Id: I0cd0407754466327147917390c578da336e61269
With the move to object storage, we no longer have predictable
hosting locations for draft builds of opendev/storyboard-webclient
changes. Switch the OAuth and CORS ACLs in the storyboard
configuration on storyboard-dev.openstack.org to allow webclient
builds hosted anywhere, as there should be nothing sensitive we need
to protect in that StoryBoard deployment.
While here, tighten up the same ACLs for production StoryBoard to
just allow its local webclient deployment, reducing the risk of
cross-site scripting attacks.
Depends-On: https://review.opendev.org/691034
Change-Id: Ie4f5eb49a864848cfa95a3e956e6dbfa122fbb1d
This is the base url that will be used for log reporting.
Depends-On: https://review.opendev.org/675655
Change-Id: Ia92a34e9ed506931e0d736ac034f60f4f7c381fc
This removes the groups servers from our inventory as well as our
manifests/modules. We don't run the groups service anymore as many
groups migrated to meetup.com independent of us and the others have
transitioned there.
Change-Id: I7cb76611e6d30e7189821923f36a38dec9ea7241
This reverts commit 0d370a285b09bd28c5b1cdfc6b89d2997f67da5d.
Fixed by https://github.com/containers/skopeo/pull/653 so safe to
merge this once a new build appears in the PPA.
Change-Id: I858eee79d084016b6b71eec46a6118d78f68cafa
Pin skopeo back to 0.1.36-1~dev~ubuntu16.04.2~ppa14 which is before
the code that changed the required capabilities, breaking the use of
skopeo from inside of bubblewrap.
Change-Id: Ibf3000d87772d02b7325315cfeed078716e0d7bf
There are many references to review.openstack.org, and while the
redirect should work, we can also go ahead and fix them.
Change-Id: I28f398796a6392a3dffea1d25cfe2ae3a36a3589
The server has been removed, remove it from inventory.
While we're here, s/graphite.openstack.org/graphite.opendev.org/'
... it's a CNAME redirect but we might as well clean up.
Change-Id: I36c951c85316cd65dde748b1e50ffa2e058c9a88
