This cleans up zuul01 as it should no longer be used at this point. We
also make the inventory groups a bit more clear that all zuul servers
are under the opendev.org domain now.
Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/790483
Change-Id: I7885fe60028fbd87688f3ae920a24bce4d1a3acd
This zuul02 instance will replace zuul01. There are a few items to
coordinate when doing an actual switch so we haven't removed zuul01 from
inventory here. In particular we need to update gearman server config
values in the zuul cluster and we need to save queues, shutdown zuul01,
then start zuul02's scheduler and restore queues there.
I believe landing this change is safe as we don't appear to start zuul
on new instances by default. Reviewers should double check this.
Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/791039
Change-Id: I524b456e494124d8293fbe8e1468de40f3800772
We are doing this so that we can cleanup the private network + floating
IP setup that the existing mirror does. Once this new mirror is up and
happy we can cname to it and then clean up the old mirror and its
networking config. We do this in order to save an IP that the current
private network router is consuming.
Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/787628
Change-Id: I50c311087c6c28726e36913c7e081f3b3d0ee049
This handles planet.openstack.org and redirects it to the
opendev.org/openstack/planet-openstack repo, where we will put a
README and the OPML file of the last state as we deprecate this
service.
Change-Id: If141aca5efbdbe60c91ceefaa4e05c98cd0ba5bb
The OpenEdge cloud has been offline for five months, initially
disabled in I4e46c782a63279d9c18ff4ba2944c15b3027114b, so go ahead
and clean up lingering references. If it is restored later, this can
be reverted fairly easily.
Depends-On: https://review.opendev.org/783989
Depends-On: https://review.opendev.org/783990
Change-Id: I544895003344bc8202363993b52f978e1c07d061
review02.opendev.org is a much larger replacement server for review01
provided by Vexxhost. It is up and running, with gerrit2 volume
attached and DNS entries.
This adds it to the staging group with no replication and a local h2
database configured for initial bringup. There's quite a bit to
consider for full migration, but this will let us start experimenting.
Change-Id: I3638a5c0c7028dcc800ada42431b75395cff0c42
With our increased ability to test in the gate, there's not much use
for review-dev any more. Remove references.
Change-Id: I97e9865e0b655cd157acf9ffa7d067b150e6fc72
This adds a dockerfile to build an opendevorg/refstack image as well as
the jobs to build and publish it.
Change-Id: Icade6c713fa9bf6ab508fd4d8d65debada2ddb30
The hound project has undergone a small re-birth and moved to
https://github.com/hound-search/hound
which has broken our deployment. We've talked about leaving
codesearch up to gitea, but it's not quite there yet. There seems to
be no point working on the puppet now.
This builds a container than runs houndd. It's an opendev specific
container; the config is pulled from project-config directly.
There's some custom scripts that drive things. Some points for
reviewers:
- update-hound-config.sh uses "create-hound-config" (which is in
jeepyb for historical reasons) to generate the config file. It
grabs the latest projects.yaml from project-config and exits with a
return code to indicate if things changed.
- when the container starts, it runs update-hound-config.sh to
populate the initial config. There is a testing environment flag
and small config so it doesn't have to clone the entire opendev for
functional testing.
- it runs under supervisord so we can restart the daemon when
projects are updated. Unlike earlier versions that didn't start
listening till indexing was done, this version now puts up a "Hound
is not ready yet" message when while it is working; so we can drop
all the magic we were doing to probe if hound is listening via
netstat and making Apache redirect to a status page.
- resync-hound.sh is run from an external cron job daily, and does
this update and restart check. Since it only reloads if changes
are made, this should be relatively rare anyway.
- There is a PR to monitor the config file
(https://github.com/hound-search/hound/pull/357) which would mean
the restart is unnecessary. This would be good in the near and we
could remove the cron job.
- playbooks/roles/codesearch is unexciting and deploys the container,
certificates and an apache proxy back to localhost:6080 where hound
is listening.
I've combined removal of the old puppet bits here as the "-codesearch"
namespace was already being used.
Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
185797a0e5e46fd0f68f7b423e79f732c8541d68 made graphite01 (old server)
accidentally do the container restart; this should be for graphite02.
Change-Id: I881ffecf9af5ee07cc3ebcf34f0e204a6389d16b
This was a host used to transition to docker run nodepool builders. That
transition has been completed for nb01.opendev.org and nb02.opendev.org
and we don't need the third x86 builder.
Change-Id: I93c7fc9b24476527b451415e7c138cd17f3fdf9f
This server is going to be our new arm64 nodepool-builder running on the
new arm64 docker images for nodepool.
Depends-On: https://review.opendev.org/750037
Change-Id: I3b46ff901eb92c7f09b79c22441c3f80bc6f9d15
I forgot in I5b7106e2263010ff353e8a1de43e73b0c0ec57e1 this is a new
mirror, which needs the LE bits setup.
Change-Id: I3109573b2b03453049a265a829445f88f8a87557
The host is review-test.opendev.org, so hostvars for
review-test.openstack.org are not so much going to do anything.
It's easier if we just ssh as root from review to gerrit2
on review-test.
review-test needs to be in letsencrypt group and have a
handler.
We need to install mysql - it's on the existing review
servers but not in ansible, it's just left over from
puppet.
The db credentials are in /root/.gerrit_db.cnf
Change-Id: I90e3c9d1b398cc16fea9f7056cfb059c7140160e
This deploys graphite from the upstream container.
We override the statsd configuration to have it listen on ipv6.
Similarly we override the ngnix config to listen on ipv6, enable ssl,
forward port 80 to 443, block the /admin page (we don't use it).
For production we will just want to put some cinder storage in
/opt/graphite/storage on the production host and figure out how to
migrate the old stats. The is also a bit of cleanup that will follow,
because we half-converted grafana01.opendev.org -- so everything can't
be in the same group till that is gone.
Testing has been added to push some stats and ensure they are seen.
Change-Id: Ie843b3d90a72564ef90805f820c8abc61a71017d
This uses the Grafana container created with
Iddfafe852166fe95b3e433420e2e2a4a6380fc64 to run the
grafana.opendev.org service.
We retain the old model of an Apache reverse-proxy; it's well tested
and understood, it's much easier than trying to map all the SSL
termination/renewal/etc. into the Grafana container and we don't have
to convince ourselves the container is safe to be directly web-facing.
Otherwise this is a fairly straight forward deployment of the
container. As before, it uses the graph configuration kept in
project-config which is loaded in with grafyaml, which is included in
the container.
Once nice advantage is that it makes it quite easy to develop graphs
locally, using the container which can talk to the public graphite
instance. The documentation has been updated with a reference on how
to do this.
Change-Id: I0cc76d29b6911aecfebc71e5fdfe7cf4fcd071a4
This was touching a file previously, but we can safely restart apache if
the certs update as this happens non concurrently with puppet updates.
Do this to ensure the cert is kept up to date.
Change-Id: I28168770258c38d13202fad48be3f61ecdc8ec4d
This autogenerates the list of ssl domains for the ssl-cert-check tool
directly from the letsencrypt list.
The first step is the install-certcheck role that replaces the
puppet-ssl_cert_check module that does the same. The reason for this
is so that during gate testing we can test this on the test
bridge.openstack.org server, and avoid adding another node as a
requirement for this test.
letsencrypt-request-certs is updated to set a fact
letsencrypt_certcheck_domains for each host that is generating a
certificate. As described in the comments, this defaults to the first
host specified for the certificate and the listening port can be
indicated (if set, this new port value is stripped when generating
certs as is not necessary for certificate generation).
The new letsencrypt-config-certcheck role runs and iterates all
letsencrypt hosts to build the final list of domains that should be
checked. This is then extended with the
letsencrypt_certcheck_additional_domains value that covers any hosts
using certificates not provisioned by letsencrypt using this
mechanism.
These additional domains are pre-populated from the openstack.org
domains in the extant check file, minus those openstack.org domain
certificates we are generating via letsencrypt (see
letsencrypt-create-certs/handlers/main.yaml). Additionally, we
update some of the certificate variables in host_vars that are
listening on port !443.
As mentioned, bridge.openstack.org is placed in the new certcheck
group for gate testing, so the tool and config file will be deployed
to it. For production, cacti is added to the group, which is where
the tool currently runs. The extant puppet installation is disabled,
pending removal in a follow-on change.
Change-Id: Idbe084f13f3684021e8efd9ac69b63fe31484606
This is to replace the puppet managed openstack.org server
Change-Id: I0e3586befd922cb56d1a0ec9c9cb650add9b225d
Depends-On: https://review.opendev.org/728314
These are to replace the puppet-based openstack.org mirrors
Depends-On: https://review.opendev.org/728308
Change-Id: Ibdce99daa514fb445f1f8389e7c052ee151057ea
New opendev.org CI mirrors for OVH regions. The old BHS1 mirror was
in the openstack.org domain, so is added new. There was an old GRA1
mirror in the opendev.org domain, so remote it and increment the
ordinal in its short hostname to avoid a collision in the inventory
cache.
This is being done to switch to un-billed flavors in this provider,
to simplify internal billing for their donation of resources.
Change-Id: I05770856b5704aa438ed6bc54ec42ba9efb5cd2a
Upstream likes building the settings file into the image, but that's
less exciting, let's bind-mount ours in.
Depends-On: https://review.opendev.org/717491/
Change-Id: Ia1894d884ef2a84e1282345b77fe07bf8898f367
We are starting over with the container nodepool host, and also moving
it to "nb04" to avoid any possibility of conflicting with the
short-hostname of nb01.openstack.org.
Story: #2007407
Task: #39064
Depends-On: https://review.opendev.org/713575
Depends-On: https://review.opendev.org/713571
Change-Id: I18ab9834ad4da201774e0abef56f618cd7839d36
Start installing Let's Encrypt SSL certs on static01 for a new
docs.airshipit.org it will be hosting.
Change-Id: Ia5089515cd02d78267827840521cbee4f71ceb0b
This adds the Open Edge (formerly Fortnebula) CI mirror.
Change-Id: I1ccf2a602f8a41e00bc64a9516a326cc07d9b254
Depends-On: https://review.opendev.org/711787
Sister change for Ia5caff34d3fafaffc459e7572a4eef6bd94422ea and
removing earlier references to the mirror server in preparation for
building and adding the new one.
Change-Id: I7d506be85326835d5e77a0c9c461f2d457b1dfd3
This site was never used nor published, it can be killed according to QA
PTL.
codesearch returns no matches for it in any docs.
Keep the occurence in manifests/static.pp, this will get deleted
as part of https://review.opendev.org/710388.
Change-Id: I3c0d3b567a3eccb959dc903f169197e4581f1e13
There is a bug, or misfeature, in acme.sh using dns manual mode where
it will not renew the certificate when new domains are added to an
existing certificate. It appears to generate the TXT record requests
correctly, but then when we renew the certificate it thinks it is not
time and skips it. This is filed upstream with [1] however we can
work around it, and generally be better anyway.
For each letsencrypt host, during certificate request we build up the
"acme_txt_required" key which is a list of TXT record tuples.
Currently we keep the challenge domain in the first entry, which is
not useful (all our hosts have the same challenge domain,
amce.opendev.org). Modify this to be the certificate key from the
host config. To be clear; when a host has
letsencrypt_certs:
hostname-cert-main:
hostname.opendev.org
altname.opendev.org
hostname-cert-secondary:
secondary.opendev.org
secondaryalt.opendev.org
acme_txt_required when renewing all certs will end up looking like:
[
(hostname-cert-main, <txt1>), (hostname-cert-main, <txt2>),
(hostname-cert-secondary, <txt3>), (hostname-cert-secondary, <txt3>>)
]
In the certificate creation path, we walk "acme_txt_required" and take
the unique 0-value entries; this gives us the list of keys in
"letsencrypt_certs" which were actually updated.
We can then force renewal for these certs, because we know they
changed in some way that requires reissuing them (within renewal time,
or new domains).
This isn't just a work-around, it is generically better too.
Previously if any cert on host required an update, we would try to
update them all. This would be a no-op; acme.sh would just skip doing
anything; but now we don't even have to call into the renewal if we
know nothing has changed.
[1] https://github.com/acmesh-official/acme.sh/issues/2763
Change-Id: I1e82c64217d46d7e1acc0111dff4db2f0062c42a
This is a slight divergence from the accepted spec, where we were
going to implement these redirects via a new haproxy instance
(I961456d44a56f2334d3c94ef27e408f27409cd65). We've decided it's
easier to keep them on static.opendev.org
The following sites are configured to redirect to whatever they are
redirecting to now on static.opendev.org:
* devstack.org
* www.devstack.org
* ci.openstack.org
* cinder.openstack.org
* glance.openstack.org
* horizon.openstack.org
* keystone.openstack.org
* nova.openstack.org
* qa.openstack.org
* summit.openstack.org
* swift.openstack.org
As a bonus, they all get a https instance too, which they didn't have
before.
testinfra coverage should be total for this change. I have created
the _acme-challange CNAME records for all the above.
Story: #2006598
Task: #38881
Change-Id: I3f1fc108e7bb1c9500ad4d1a51df13bb4ae00cb9
This creates the redirect sites
git.airshipit.org
git.openstack.org
git.starlingx.io
git.zuul-ci.org
The htaccess rules are put into the main configuration file to avoid
having to create a directory and manage another file. We use a macro
to duplicate the rules and retain the old semantics of the http site
redirecting directly (as opposed to doing a extra 301 to
https://git.openstack.org first). This required adding "/" to the "^"
matches as it now runs in VirtualHost context; no functional change is
intended over the old sites.
This will require _acme-challenge CNAMEs to acme.opendev.org before
being merged.
testinfra is updated to exercise some redirects matching against the
results of the extant sites.
Change-Id: Iaa9d5dc2af3f5f8abc11c2312e4308b50f5fcd2b
This creates sites to serve
developer.openstack.org
docs.openstack.org
docs.opendev.org
docs.starlingx.io
which are all just static directories underneath /afs/openstack.org/.
This is currently done by files02.openstack.org, but will be better
served in the future by consolidating in ansible configuration on
static.opendev.org.
The following dns entries need to be made before merging to ensure the
certificates are provisioned
_acme-challenge.developer.openstack.org
_acme-challenge.docs.openstack.org
_acme-challenge.docs.opendev.org
_acme-challenge.docs.starlingx.io
Once done, we can merge and then cut-over the main DNS entries as we
like.
Since there are some follow-ons, I have not removed the puppet
configuration from files02.openstack.org. I think it's best we
migrate everything away from that and remove it in one lot.
Change-Id: I459a36f823a8868e6cc09e2b0d85f2fe05d69002
This adds the site to publish from
/afs/openstack.org/project/releases.openstack.org
Change-Id: Ia91deb9a51441ac9974137ed39fc5a185689a11c
Task: #37724
Story: #2006598
