This change contains the roles and testing for deploying certificates
on hosts using letsencrypt with domain authentication.
From a top level, the process is implemented in the roles as follows:
1) letsencrypt-acme-sh-install
This role installs the acme.sh tool on hosts in the letsencrypt
group, along with a small custom driver script to help parse output
that is used by later roles.
2) letsencrypt-request-certs
This role runs on each host, and reads a host variable describing
the certificates required. It uses the acme.sh tool (via the
driver) to request the certificates from letsencrypt. It populates
a global Ansible variable with the authentication TXT records
required.
If the certificate exists on the host and is not within the renewal
period, it should do nothing.
3) letsencrypt-install-txt-record
This role runs on the adns server. It installs the TXT records
generated in step 2 to the acme.opendev.org domain and then
refreshes the server. Hosts wanting certificates will have
pre-provisioned CNAME records for _acme-challenge.host.opendev.org
pointing to acme.opendev.org.
4) letsencrypt-create-certs
This role runs on each host, reading the same variable as in step
2. However this time the acme.sh tool is run to authenticate and
create the certificates, which should now work correctly via the
TXT records from step 3. After this, the host will have the
full certificate material.
Testing is added via testinfra. For testing purposes requests are
made to the staging letsencrypt servers and a self-signed certificate
is provisioned in step 4 (as the authentication is not available
during CI). We test that the DNS TXT records are created locally on
the CI adns server, however.
Related-Spec: https://review.openstack.org/587283
Change-Id: I1f66da614751a29cc565b37cdc9ff34d70fdfd3f
This adds the concept of an unmanaged domain; for unmanaged domains we
will write out the zone file only if it doesn't already exist.
acme.opendev.org is added as an unmanaged domain. It will be managed
by other ansible roles which add TXT records for ACME authentication.
The initial template comes from the dependent change, and this ensures
the bind configuration is always valid.
For flexibility and testing purposes, we allow passing an extra
refspec and version to the git checkout. This is one way to pull in
changes for speculative CI runs (I looked into having the hosts under
test checkout from Zuul; but by the time we're 3-ansible call's deep
on the DNS hosts-under-test it's a real pain. For the amount of times
we update this, it's easier to just allow a speculative change that
can take a gerrit URL; for an example see [1])
[1] https://review.openstack.org/#/c/641155/10/playbooks/group_vars/dns.yaml
Testing is enhanced to check for zone files and correct configuration
stanzas.
Depends-On: https://review.openstack.org/641154
Depends-On: https://review.openstack.org/641168
Change-Id: I9ef5cfc850c3458c63aff46cfaa0d49a5d194e87
There's no real need to tie these together into a multi-stage
Dockerfile as they don't really share anything. Split them.
Change-Id: Ifd7ccadcd8048eeb57797d60356aec2f9f0d2c80
Depends-On: https://review.openstack.org/641805
When we run the run-gitea job, make sure that it runs *after*
the gitea image build job, if the build job ran at all. If
it didn't, we don't need to worry about it.
This has to happen in the project pipeline config because the
job dependency is different in check and gate.
Change-Id: I7cb069a6cd40a4ae8d5cbcdf23e7686b301493a1
These now only run a registry if there isn't already one running
in the buildset, so we can use them here in that configuration.
They also take care of the provides/requires for us, so we can
remove those.
Change-Id: I3b6d825df8d658adcae51ed20a5335c612b7d9f2
Depends-On: https://review.openstack.org/642151
The web page from which this call is taken has a readme selector
and a default value of "Default" that we're not sending in this
request. Send it to avoid gitea not being able to find the empty
readme.
Also, add gitea-git-repos to the files section of system-config-run-gitea
so that we actually test it.
Change-Id: Ieec94aadb63fa097f10a3f325dd105b30e610dd9
Add an option to run a playbook (in the fake bridge context) after
running the base playbook. Use this to run a new playbook which
exercises gitea project creation after bootstrapping the gitea
service.
Disable ansible-lint 304 because it erroneously thinks shell and
command are the same thing.
Change-Id: I0394b614771bc62b9fe23d811defd7767b3d10db
Now that nodepool and zuul support provider affinity for paused
jobs, we should be able to have a buildset registry run for
every change and configure the docker image based jobs to use it.
Change-Id: Iddf12475c8dfce2db2d79538ace58a49211eab97
This runs an haproxy which is strikingly similar to the one we
currently run for git.openstack.org, but it is run in a docker
container.
Change-Id: I647ae8c02eb2cd4f3db2b203d61a181f7eb632d2
Also, correct the host_vars filename. Again.
Also, make sure we run the test on changes to the host_vars filename.
Change-Id: I95fb61531bae677f5c68f4e56ed718da6c507eb9
There are upstream jobs in zuul-jobs with the docker build playbooks,
so use them. The system-config jobs are kept so that we don't have
to duplicate the secret stanza.
Change-Id: Iceee55a3d0e8b243549fa988f134b1ea9bb6dac5
We've made separate dockerhub users so we don't have to share
credentials across trust boundaries, nor do we need to define the
jobs which use them centrally.
There is a new opendevzuul user which has access to opendevorg,
use that.
Change-Id: Ia22188255f6e5b0f2ba0cefe20694bdec38c1444
This is a builder image intended to make it easy to build
images for projects that use pbr and bindep.
Also, build a python-base image for use with python-builder
pbrx installed dumb-init in a base image and used it as an
entrypoint for the image. This is a good pattern that we should
continue. Make a python-base image that can be used in the FROM
line of consuming final images.
Change-Id: I8da39e680e9c5ca0d8f603a2645e83fc3be87a8b
The upstream role var names have changed. For the better.
Depends-On: https://review.openstack.org/631602
Change-Id: I0a338da98dcfb48cd4207f182f88a043b1370ab6
Without this, other projects could run this job (or its descendents)
and pass in an images dictionary instructing it to upload something
to one of our repositories.
Change-Id: I2c68d6673217bbc274c1134ee221cd6484abcf16
Add jobs to build, upload, and promote a gerrit image to
Docker Hub. This uses alternative tags, build path, and
build args.
Change-Id: Iea7109f9ee0ec18431f298a2ecc611678a868d6d
Co-Authored-By: Monty Taylor <mordred@inaugust.com>
Add jobs to build, upload, and promote a gitea image to Docker Hub.
This produces two images (gitea and openssh) from a single Dockerfile
via a multi-stage build.
Change-Id: I8e6136787f75cd69b881b7ac715418c86d356872
This adds the infrastructure for building docker images: the
credential used to upload to Docker Hub as well as the parent jobs
and playbooks to perform the builds.
Change-Id: I7cbbcdd184c4934f1b0ce5905d9760c732b06aa9
Depends-On: https://review.openstack.org/631078
The gerrit source dir needs three plugins cloned into
the plugins dir and also a few files updated.
Depends-On: https://review.openstack.org/631007
Change-Id: I56037137d43ee1cea0a4c17e48d09102e1599ddc
If we clone gerrit to ~/src/gerrit.googlesource.com/gerrit but
want to keep the Dockerfile in system-config, then we need to be
able to run:
docker build ~/src/gerrit.googlesource.com/gerrit -f Dockerfile
Most of the time the dir will just be '.', so put in a sensible
default.
Change-Id: I235080c05e679d2ac270cd5401b85c655fab3112
The credentials secret needs to be associated with the playbook
that uses it.
Also, add build-image-jinja-init to check, this was accidentally
omitted earlier.
Change-Id: I527a26781a716f9307539e4a486d6b79b32f25fd
Upload an image to dockerhub with a change-specific tag in every
gate job, and then, if the change lands, re-tag the image in
dockerhub.
Change-Id: Ie57fc342cbe29d261d33845829b77a0c1bae5ff4
This creates a jinja-init image based on
https://github.com/ObjectifLibre/jinja-init
We can use this image as an "init container" in k8s to template out
config files using configmaps and secrets.
It also adds a gitea-init Dockerfile, which will require the
jinja-init image to be published first.
Change-Id: I88e14c4f0d868f99aa0a0316e6b9e24c2a8fea68
This is a role for installing docker on our control-plane servers.
It is based on install-docker from zuul-jobs.
Basic testinfra tests are added; because docker fiddles the iptables
rules in magic ways, the firewall testing is moved out of the base
tests and modified to partially match our base firewall configuration.
Change-Id: Ia4de5032789ff0f2b07d4f93c0c52cf94aa9c25c
This change takes the ARA report from the "inner" run of the base
playbooks on our bridge.o.o node and publishes it into the final log
output. This is then displayed by the middleware.
Create a new log hierarchy with a "bridge.o.o" to make it clear the
logs here are related to the test running on that node. Move the
ansible config under there too.
Change-Id: I74122db09f0f712836a0ee820c6fac87c3c9c734
Due to an issue with the current ansible branch, this check is failing
currently, make it non-voting to unblock other changes.
[0] https://github.com/ansible/ansible/issues/49430
Change-Id: I2fcf3abd69103a3e0b8eb56528650481a865a1cc
Similar to the pinning introduced in
Ic465efb637c0a1eb475f04b0b0e356d8797ecdeb, use the "latest"
openstacksdk package and allow for passing of pinned versions if
required.
Update the devel test to also use the master of opensatcksdk
Change-Id: I4b437ca9024c87903bdd3569c8309cde725ce28e
This adds arguments to "install-ansible" to allow us to specify the
package name and version.
This is used to pin bridge.o.o to 2.7.0 (see
I9cf4baf1b15893f0c677567f5afede0d0234f0b2).
A new job is added to test against the ansible-devel branch. Added as
voting for now, until it proves to be a concern.
Change-Id: Ic465efb637c0a1eb475f04b0b0e356d8797ecdeb
This new job is a parent job allowing us to CD from Zuul via
bridge.openstack.org. Using Zuul project ssh keys we add_host bridge.o.o
to our running inventory on the executor then run ansible on bridge.o.o
to run an ansible playbook in
bridge.openstack.org:/opt/system-config/playbooks.
Change-Id: I5cd2dcc53ac480459a22d9e19ef38af78a9e90f7
It's very convenient to be able to temporarily stop running all
these heavyweight jobs when developing new jobs. Also, the config
is currently split in a strange way. Move it all in-repo for
convenience.
Change-Id: I4da96050f1a20438ca29454de33cd871f5b282c8
Create a parent run job and inherit from it. This reduces duplicate
'run' parameters, and corrects the omission of run-post from the
eavesdrop job.
Change-Id: Ib2a21b7190bf3611972097d6db545989cd54b3d4
Add a job which runs testinfra for the eavesdrop server. When we
have a per-hostgroup playbook, we will add it to this job too.
The puppet group is removed from the run-base job because the
groups.yaml file is now used to construct groups (as it does
in production) and will construct the group correctly.
The testinfra iptables module may throw an error if it's run
multiple times simultaneously on the same host. To avoid this,
stop using parallel execution.
Change-Id: I1a7bab5c14b0da22393ab568000d0921c28675aa
