This includes a switch from the "legacy" style Wildfly-based image
to a new setup using Quarkus.
Because Keycloak maintainers consider H2 databases as a test/dev
only option, there are no good migration and upgrade paths short of
export/import data. Go ahead and change our deployment model to rely
on a proper RDBMS, run locally from a container on the same server.
Change-Id: I01f8045563e9f6db6168b92c5a868b8095c0d97b
We often need to update gerrit image build details that only live in the
job specification. For example tag or branch versions of gerrit and
related repos. When we do this if we don't also do a noop update to our
Dockerfiles the promotion job doesn't run for these images because we
can't do the implicit file match in the promotion (deploy) pipeline.
Fix this by explicitly matching the job config file in our jobs so that
when we update the gerrit jobs we also run the gerrit image promotion
jobs.
We also ensure the system-config-run-review and
infra-prod-service-review jobs are triggered when the docker image jobs
update. This ensures we actually test the resulting images and then
perform potentially necessary deployment actions before they are pulled
into use.
Change-Id: Id0c51818cd1e01bd16a79ab0c0f9172e844376b8
Now that the Mailman v3 migration is complete, we no longer need any
divergence between the lists01 (production) and lists99 (test node)
host vars, so put everything into the group vars file instead.
Change-Id: If92943694e95ef261fbd254eff65a51d8d3f7ce5
Clean up references to lists.openstack.org other than as a virtual
host on the new lists01.opendev.org Mailman v3 server. Update a few
stale references to the old openstack-infra mailing list (and
accompanying stale references to the OpenStack Foundation and
OpenStack Infra team). Update our mailing list service documentation
to reflect the new system rather than the old one. Once this change
merges, we can create an archival image of the old server and delete
it (as well as removing it from our emergency skip list for
Ansible).
Side note, the lists.openstack.org server will be 11.5 years old on
November 1, created 2012-05-01 21:14:53 UTC. Farewell, old friend!
Change-Id: I54eddbaaddc7c88bdea8a1dbc88f27108c223239
Fix the infra-prod-service-lists3 job to trigger when we update the
mailman3.yaml group vars file. In addition we make a noop reorganization
change to the mailman3 group file to group exim vars together which will
be used to ensure that this change triggers the lists3 job as expected.
In system-config-run-lists3 we update that job to be triggered when we
update the docker images for mailman. We don't bother testing this now
as that would be masked off by the update to the mailman3 groups file.
But in the future when we do mailman3 image updates we'll be looking for
this job to run.
Change-Id: I994b0a79bf46f525dd9e059719f5a08c9c390b8c
This uncomments the list additions for the lists.airshipit.org and
lists.katacontainers.io sites on the new mailman server, removing
the configuration for them from the lists.opendev.org server and, in
the case of the latter, removing all our configuration management
for the server as it was the only site hosted there.
Change-Id: Ic1c735469583e922313797f709182f960e691efc
Add the docker/mailman tree to the infra-prod-service-lists3 job so
that we deploy new versions whenever we make changes to the
container images.
Change-Id: Ife5e878b1f81c2879c2959fe6d4de22fe841583b
We are currently using the default nodeset on the
infra-prod-bootstrap-bridge job which results in us waiting for a node
that we end up ignoring. As far as I can tell this job runs against
localhost and the add_host bridge entry. It ignores the default test
node from the nodeset.
Speed up job execution and reduce node waste by setting an empty nodeset
on the job.
Change-Id: I8c3ffda60b92a8655989579335a49423fbdd18a2
The gitea and static services both deploy apache using our UA filters.
We were not testing these services when updating UA filters. This change
fixes that giving us some basic sanity checking that UA filter updates
are functional and quicker deployments of these filters.
Change-Id: Icbe6558bb47946299a43905e2f64522576bad939
This switches us to running the services against the etherpad group. We
also define vars in a group_vars file rather than a host specific
file. This allows us to switch testing over to etherpad99 to decouple it
from our production hostnames.
A followup change will add a new etherpad production server that will be
deployed alongside the existing one. This refactor makes that a bit
simpler.
Change-Id: I838ad31eb74a3abfd02bbfa77c9c2d007d57a3d4
We have access to manage the linaro cloud, but we don't want to
completely own the host as it has been configured with kolla-ansible;
so we don't want to take over things like name resolution, iptables
rules, docker installation, etc.
But we would like to manage some parts of it, like rolling out our
root users, some cron jobs, etc. While we could just log in and do
these things, it doesn't feel very openinfra.
This allows us to have a group "unmanaged" that skips the base jobs.
The base playbook is updated to skip these hosts.
For now, we add a cloud-linaro prod job that just does nothing so we
can validate the whole thing. When it's working, I plan to add a few
things as discussed above.
Change-Id: Ie8de70cbac7ffb9d727a06a349c3d2a3b3aa0b40
Firstly, my understanding of "adns" is that it's short for
authoritative-dns; i.e. things related to our main non-recursive DNS
servers for the zones we manage. The "a" is useful to distinguish
this from any sort of other dns services we might run for CI, etc.
The way we do this is with a "hidden" server that applies updates from
config management, which then notifies secondary public servers which
do a zone transfer from the primary. They're all "authoritative" in
the sense they're not for general recursive queries.
As mentioned in Ibd8063e92ad7ff9ee683dcc7dfcc115a0b19dcaa, we
currently have 3 groups
adns : the hidden primary bind server
ns : the secondary public authoratitive servers
dns : both of the above
This proposes a refactor into the following 3 groups
adns-primary : hidden primary bind server
adns-secondary : the secondary public authoritative servers
adns : both of the above
This is meant to be a no-op; I just feel like this makes it a bit
clearer as to the "lay of the land" with these servers. It will need
some considering of the hiera variables on bridge if we merge.
Change-Id: I9ffef52f27bd23ceeec07fe0f45f9fee08b5559a
We changed review01.openstack.org to review02.openstack.org in the host
var file matchers for this job thinking that was the issue previously.
Unfortunately the actual file is review02.opendev.org. Update the
matcher again to actually trigger the job.
We also make a small edit to the gerrit role's README to ensure we
trigger the job when this change lands.
Change-Id: I1f235d0ddbb2d7f400ea2e99ffabdf5db35671a1
We didn't update this job's file matchers when review01 was replaced
with review02. That caused us to miss triggering this job when review02
hostvars updated. Fix that which should also cause this job to run since
we update the job.
Change-Id: I8b58ee26084681242b9881651d6eeab9ff8d5ad2
We need the infra-prod-bootstrap-bridge job to add SSH host keys
from our Ansible inventory to the /etc/ssh_known_hosts on the
bridge. When adding a new server to the inventory, any added host
keys should be deployed. Make sure this happens.
Change-Id: I422f80fc033cfe8e20d6d30b0fe23f82800c4cea
This should now be a largely functional deployment of mailman 3. There
are still some bits that need testing but we'll use followup changes to
force failure and hold nodes.
This deployment of mailman3 uses upstream docker container images. We
currently hack up uids and gids to accomodate that. We also hack up the
settings file and bind mount it over the upstream file in order to use
host networking. We override the hyperkitty index type to xapian. All
list domains are hosted in a single installation and we use native
vhosting to handle that.
We'll deploy this to a new server and migrate one mailing list domain at
a time. This will allow us to start with lists.opendev.org and test
things like dmarc settings before expanding to the remaining lists.
A migration script is also included, which has seen extensive
testing on held nodes for importing copies of the production data
sets.
Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
Following-on from Iffb462371939989b03e5d6ac6c5df63aa7708513, instead
of directly referring to a hostname when adding the bastion host to
the inventory for the production playbooks, this finds it from the
first element of the "bastion" group.
As we do this twice for the run and post playbooks, abstract it into a
role.
The host value is currently "bridge.openstack.org" -- as is the
existing hard-coding -- thus this is intended to be a no-op change.
It is setting the foundation to make replacing the bastion host a
simpler process in the future.
Change-Id: I286796ebd71173019a627f8fe8d9a25d0bfc575a
This replaces hard-coding of the host "bridge.openstack.org" with
hard-coding of the first (and only) host in the group "bastion".
The idea here is that we can, as much as possible, simply switch one
place to an alternative hostname for the bastion such as
"bridge.opendev.org" when we upgrade. This is just the testing path,
for now; a follow-on will modify the production path (which doesn't
really get speculatively tested)
This needs to be defined in two places :
1) We need to define this in the run jobs for Zuul to use in the
playbooks/zuul/run-*.yaml playbooks, as it sets up and collects
logs from the testing bastion host.
2) The nested Ansible run will then use inventory
inventory/service/groups.yaml
Various other places are updated to use this abstracted group as the
bastion host.
Variables are moved into the bastion group (which only has one host --
the actual bastion host) which means we only have to update the group
mapping to the new host.
This is intended to be a no-op change; all the jobs should work the
same, but just using the new abstractions.
Change-Id: Iffb462371939989b03e5d6ac6c5df63aa7708513
In discussion of other changes, I realised that the bridge bootstrap
job is running via zuul/run-production-playbook.yaml. This means it
uses the Ansible installed on bridge to run against itself -- which
isn't much of a bootstrap.
What should happen is that the bootstrap-bridge.yaml playbook, which
sets up ansible and keys on the bridge node, should run directly from
the executor against the bridge node.
To achieve this we reparent the job to opendev-infra-prod-setup-keys,
which sets up the executor to be able to log into the bridge node. We
then add the host dynamically and run the bootstrap-bridge.yaml
playbook against it.
This is similar to the gate testing path; where bootstrap-bridge.yaml
is run from the exeuctor against the ephemeral bridge testing node
before the nested-Ansible is used.
The root key deployment is updated to use the nested Ansible directly,
so that it can read the variable from the on-host secrets.
Change-Id: Iebaeed5028050d890ab541818f405978afd60124
Similar to Id98768e29a06cebaf645eb75b39e4dc5adb8830d, move the
certificate variables to the group definition file, so that we don't
have to duplicate handlers or definitions for the testing host.
Change-Id: I6650f5621a4969582f40700232a596d84e2b4a06
Currently we define the letsencrypt certs for each host in its
individual host variables.
With recent work we have a trusted CA and SAN names setup in
our testing environment; introducing the possibility that we could
accidentally reference the production host during testing (both have
valid certs, as far as the testing hosts are concerned).
To avoid this, we can use our naming scheme to move our testing hosts
to "99" and avoid collision with the production hosts. As a bonus,
this really makes you think more about your group/host split to get
things right and keep the environment as abstract as possible.
One example of this is that with letsencrypt certificates defined in
host vars, testing and production need to use the same hostname to get
the right certificates created. Really, this should be group-level
information so it applies equally to host01 and host99. To cover
"hostXX.opendev.org" as a SAN we can include the inventory_hostname in
the group variables.
This updates one of the more tricky hosts, static, as a proof of
concept. We rename the handlers to be generic, and update the testing
targets.
Change-Id: Id98768e29a06cebaf645eb75b39e4dc5adb8830d
This reverts commit 21c6dc02b5b3069e4c9410416aeae804b2afbb5c.
Everything appears to be working with Ansible 2.9, which does seem to
sugguest reverting this will result in jobs timing out again. We will
monitor this, and I76ba278d1ffecbd00886531b4554d7aed21c43df is a
potential fix for this.
Change-Id: Id741d037040bde050abefa4ad7888ea508b484f6
We've been seeing ansible post-run playbook timeouts in our infra-prod
jobs. The only major thing that has changed recently is the default
update to ansible 5 for these jobs. Force them back to 2.9 to see if the
problem goes away.
Albin Vass has noted that there are possibly glibc + debian bullseye +
ansible 5 problems that may be causing this. If we determine 2.9 is
happy then this is the likely cause.
Change-Id: Ibd40e15756077d1c64dba933ec0dff6dc0aac374
If the production playbook times out, we don't get any logs collected
with the run. By moving the log collection into a post-run step, we
should always get something copied to help us diagnose what is going
wrong.
Change-Id: I3e99b80e442db0cc87f8e8c9728b7697a5e4d1d3
This was running on all group var updates but we only need to run it
when refstack group vars update. Change the file requirements to match
the refstack.yaml group file to address this.
Change-Id: Id5ed4b65c1ed6566696fea9a33db27e9318af1a6
We have validated that the log encryption/export path is working, so
turn it on for all prod jobs.
Change-Id: Ic04d5b6e716dffedc925cb799e3630027183d890
Based on the changes in I5b9f9dd53eb896bb542652e8175c570877842584,
enable returning encrypted log artifacts for the codesearch production
job, as an initial test.
Change-Id: I9bd4ed0880596968000b1f153c31df849cd7fa8d
This used to be called "bridge", but was then renamed with
Ia7c8dd0e32b2c4aaa674061037be5ab66d9a3581 to install-ansible to be
clearer.
It is true that this is installing Ansible, but as part of our
reworking for parallel jobs this is the also the synchronisation point
where we should be deploying the system-config code to run for the
buildset.
Thus naming this "boostrap-bridge" should hopefully be clearer again
about what's going on.
I've added a note to the job calling out it's difference to the
infra-prod-service-bridge job to hopefully also avoid some of the
inital confusion.
Change-Id: I4db1c883f237de5986edb4dc4c64860390cc8e22
This adds a keycloak server so we can start experimenting with it.
It's based on the docker-compose file Matthieu made for Zuul
(see https://review.opendev.org/819745 )
We should be able to configure a realm and federate with openstackid
and other providers as described in the opendev auth spec. However,
I am unable to test federation with openstackid due its inability to
configure an oauth app at "localhost". Therefore, we will need an
actual deployed system to test it. This should allow us to do so.
It will also allow use to connect realms to the newly available
Zuul admin api on opendev.
It should be possible to configure the realm the way we want, then
export its configuration into a JSON file and then have our playbooks
or the docker-compose file import it. That would allow us to drive
change to the configuration of the system through code review. Because
of the above limitation with openstackid, I think we should regard the
current implementation as experimental. Once we have a realm
configuration that we like (which we will create using the GUI), we
can chose to either continue to maintain the config with the GUI and
appropriate file backups, or switch to a gitops model based on an
export.
My understanding is that all the data (realms configuration and session)
are kept in an H2 database. This is probably sufficient for now and even
production use with Zuul, but we should probably switch to mariadb before
any heavy (eg gerrit, etc) production use.
This is a partial implementation of https://docs.opendev.org/opendev/infra-specs/latest/specs/central-auth.html
We can re-deploy with a new domain when it exists.
Change-Id: I2e069b1b220dbd3e0a5754ac094c2b296c141753
Co-Authored-By: Matthieu Huin <mhuin@redhat.com>
Having two groups here was confusing. We seem to use the review group
for most ansible stuff so we prefer that one. We move contents of the
gerrit group_vars into the review group_vars and then clean up the use
of the old group vars file.
Change-Id: I7fa7467f703f5cec075e8e60472868c60ac031f7
Previously we had a test specific group vars file for the review Ansible
group. This provided junk secrets to our test installations of Gerrit
then we relied on the review02.opendev.org production host vars file to
set values that are public.
Unfortunately, this meant we were using the production heapLimit value
which is far too large for our test instances leading to the occasionaly
failure:
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 9596567552 bytes for committing reserved memory.
We cannot set the heapLimit in the group var file because the hostvar
file overrides those values. To fix this we need to replace the test
specific group var contents with a test specific host var file instead.
To avoid repeating ourselves we also create a new review.yaml group_vars
file to capture common settings between testing and prod. Note we should
look at combining this new file with the gerrit.yaml group_vars.
On the testing side of things we set the heapLimit to 6GB, we change the
serverid value to prevent any unexpected notedb confusion, and we remove
replication config.
Change-Id: Id8ec5cae967cc38acf79ecf18d3a0faac3a9c4b3
This bumps the gerrit image up to our 3.3 image. Followup changes will
shift upgrade testing to test 3.3 to 3.4 upgrades, clean up no longer
needed 3.2 images, and start building 3.4 images.
Change-Id: Id0f544846946d4c50737a54ceb909a0a686a594e
We have a subdir in inventory called base that includes the shared
files that we don't have a good way to distinguish between services.
Limit the file matchers to inventory/base so that we don't trigger
all of the services anytime a single service's host_vars changes.
Change-Id: I3f461b4ab56ec55beca29e123186b36513803a44
This order is important to ensure we update the matrix eavesdrop bot
when expected and not later in the day when the daily runs happen.
Change-Id: If8e3f9f34e30cdeb7765e6665d1fb19b339454a3
These services are all managed with ansible now and don't need to be
triggered when puppet updates.
Change-Id: Ie32b788263724ad9a5ca88a6406290309ec8c87a
Update the file matchers to actually match the current set of puppet
things. This ensure the deploy job runs when we want it and we can catch
up daily instead of hourly.
Previously a number of the matchers didn't actually match the puppet
things because the path prefix was wrong or works were in different
orders for the dir names.
Change-Id: I3510da81d942cf6fb7da998b8a73b0a566ea7411
