opendev/system-config
Code Issues Proposed changes
d3875c1867
system-config/zuul.d/project.yaml

655 lines
27 KiB
YAML
Raw Normal View History

Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- project:
templates:
- system-config-zuul-role-integration
- system-config-gerrit-images
check:
jobs:
- opendev-tox-docs
Run opendev-buildset-regitry on Noble This unpins the nodeset for opendev-buildset-registry which should start working once the parent change has landed and our new mirror content has been copied and deployed. We'll need to recheck this change when we believe we've reached that state. Change-Id: I7a92036cc63963d3066b4eb2bf451dd0ec8a887b
2024-08-21 16:43:16 -07:00
- opendev-buildset-registry
Set tox-linters job timeout to the default This job only takes a few minutes to run. There isn't any reason to use an hour long timeout on this job as a result. Reset it back to the default and simplify the config. Change-Id: Id808e2fa35caa7cf627014b04d1560326eeebc5b
2024-08-22 13:10:26 -07:00
- tox-linters
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-run-base
- system-config-run-base-ansible-devel:
voting: false
Add borg-backup roles This adds roles to implement backup with borg [1]. Our current tool "bup" has no Python 3 support and is not packaged for Ubuntu Focal. This means it is effectively end-of-life. borg fits our model of servers backing themselves up to a central location, is well documented and seems well supported. It also has the clarkb seal of approval :) As mentioned, borg works in the same manner as bup by doing an efficient back up over ssh to a remote server. The core of these roles are the same as the bup based ones; in terms of creating a separate user for each host and deploying keys and ssh config. This chooses to install borg in a virtualenv on /opt. This was chosen for a number of reasons; firstly reading the history of borg there have been incompatible updates (although they provide a tool to update repository formats); it seems important that we both pin the version we are using and keep clients and server in sync. Since we have a hetrogenous distribution collection we don't want to rely on the packaged tools which may differ. I don't feel like this is a great application for a container; we actually don't want it that isolated from the base system because it's goal is to read and copy it offsite with as little chance of things going wrong as possible. Borg has a lot of support for encrypting the data at rest in various ways. However, that introduces the possibility we could lose both the key and the backup data. Really the only thing stopping this is key management, and if we want to go down this path we can do it as a follow-on. The remote end server is configured via ssh command rules to run in append-only mode. This means a misbehaving client can't delete its old backups. In theory we can prune backups on the server side -- something we could not do with bup. The documentation has been updated but is vague on this part; I think we should get some hosts in operation, see how the de-duplication is working out and then decide how we want to mange things long term. Testing is added; a focal and bionic host both run a full backup of themselves to the backup server. Pretty cool, the logs are in /var/log/borg-backup-<host>.log. No hosts are currently in the borg groups, so this can be applied without affecting production. I'd suggest the next steps are to bring up a borg-based backup server and put a few hosts into this. After running for a while, we can add all hosts, and then deprecate the current bup-based backup server in vexxhost and replace that with a borg-based one; giving us dual offsite backups. [1] https://borgbackup.readthedocs.io/en/stable/ Change-Id: I2a125f2fac11d8e3a3279eb7fa7adb33a3acaa4e
2020-07-16 13:43:18 +10:00
- system-config-run-borg-backup
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-run-dns
- system-config-run-eavesdrop:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-accessbot
soft: true
limnoria/meetbot setup on eavesdrop01.opendev.org This installs our Limnoira/meetbot container and configures it on eavesdrop01.opendev.org. I have ported the configuration from the old puppet as best I can (it is very verbose); my procedure was to use the Limnoira wizard to start a new config file then backport everything from the old file. I felt this was best to not miss any new options. This does channel logging (via built-in ChannelLogger plugin, along with a cron job for logs2html) and runs our fork of meetbot. It exports the channel logs via HTTP to /irclogs and meetings logs to /meetings. meetings.opendev.org will proxy to these two locations when the server is active. Note this has not ported the channel list; so the bot will not be listening in our channels. Change-Id: I9f9a466c271e1a706f9f98f816de0e84047519f1
2021-05-31 14:47:07 +10:00
- name: system-config-build-image-ircbot
soft: true
Run matrix-eavesdrop on eavesdrop Thin runs the new matrix-eavesdrop bot on the eavesdrop server. It will write logs out to the limnoria logs directory, which is mounted inside the container. Change-Id: I867eec692f63099b295a37a028ee096c24109a2e
2021-07-09 18:39:59 -07:00
- name: system-config-build-image-matrix-eavesdrop
soft: true
Migrate codesearch site to container The hound project has undergone a small re-birth and moved to https://github.com/hound-search/hound which has broken our deployment. We've talked about leaving codesearch up to gitea, but it's not quite there yet. There seems to be no point working on the puppet now. This builds a container than runs houndd. It's an opendev specific container; the config is pulled from project-config directly. There's some custom scripts that drive things. Some points for reviewers: - update-hound-config.sh uses "create-hound-config" (which is in jeepyb for historical reasons) to generate the config file. It grabs the latest projects.yaml from project-config and exits with a return code to indicate if things changed. - when the container starts, it runs update-hound-config.sh to populate the initial config. There is a testing environment flag and small config so it doesn't have to clone the entire opendev for functional testing. - it runs under supervisord so we can restart the daemon when projects are updated. Unlike earlier versions that didn't start listening till indexing was done, this version now puts up a "Hound is not ready yet" message when while it is working; so we can drop all the magic we were doing to probe if hound is listening via netstat and making Apache redirect to a status page. - resync-hound.sh is run from an external cron job daily, and does this update and restart check. Since it only reloads if changes are made, this should be relatively rare anyway. - There is a PR to monitor the config file (https://github.com/hound-search/hound/pull/357) which would mean the restart is unnecessary. This would be good in the near and we could remove the cron job. - playbooks/roles/codesearch is unexciting and deploys the container, certificates and an apache proxy back to localhost:6080 where hound is listening. I've combined removal of the old puppet bits here as the "-codesearch" namespace was already being used. Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
2020-11-17 17:13:46 +11:00
- system-config-run-codesearch:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-hound
soft: true
kerberos-kdc: role to manage Kerberos KDC servers This adds a role and related testing to manage our Kerberos KDC servers, intended to replace the puppet modules currently performing this task. This role automates realm creation, initial setup, key material distribution and replica host configuration. None of this is intended to run on the production servers which are already setup with an active database, and the role should be effectively idempotent in production. Note that this does not yet switch the production servers into the new groups; this can be done in a separate step under controlled conditions and with related upgrades of the host OS to Focal. Change-Id: I60b40897486b29beafc76025790c501b5055313d
2021-03-05 16:10:01 +11:00
- system-config-run-kerberos
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
- system-config-run-lists3:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-mailman
soft: true
Add zookeeper-statsd This adds a program, zookeeper-statsd, which monitors zookeeper metrics and reports them to statsd. It also adds a container to run that program. And it runs the container on each of the ZooKeeper quorum members. And it updates the graphite host to allow statsd traffic from quorum members. And it updates the 4-letter-word whitelist to allow the mntr command (which is used to gather metrics) to be issued. Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 12:57:21 -07:00
- system-config-run-nodepool:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-zookeeper-statsd
soft: true
Use upstream jitsi-meet web image This has our change to open etherpad on join, so we should no longer need to run a fork of the web server. Switch to the upstream container image and stop building our own. Change-Id: I3e8da211c78b6486a3dcbd362ae7eb03cc9f5a48
2021-03-02 12:59:37 -08:00
- system-config-run-meetpad
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-run-mirror-x86
- system-config-run-mirror-update
Add paste service The paste service needs an upgrade; since others have created a lodgeit container it seems worth us keeping the service going if only to maintain the historical corpus of pastes. This adds the ansible to deploy lodgeit and a sibling mariadb container. I have imported a dump of the old data as a test. The dump is ~4gb and imported it takes up about double that; certainly nothing we need to be too concerned over. The server will be more than capable of running the db container alongside the lodgeit instance. This should have no effect on production until we decide to switch DNS. Change-Id: I284864217aa49d664ddc3ebdc800383b2d7e00e3
2021-06-29 11:57:10 +10:00
- system-config-run-paste:
dependencies:
- name: opendev-buildset-registry
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-run-static
- system-config-run-docker-registry
- system-config-run-etherpad:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-etherpad
soft: true
- system-config-run-gitea:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-gitea
soft: true
- name: system-config-build-image-haproxy-statsd
soft: true
Grafana container deployment This uses the Grafana container created with Iddfafe852166fe95b3e433420e2e2a4a6380fc64 to run the grafana.opendev.org service. We retain the old model of an Apache reverse-proxy; it's well tested and understood, it's much easier than trying to map all the SSL termination/renewal/etc. into the Grafana container and we don't have to convince ourselves the container is safe to be directly web-facing. Otherwise this is a fairly straight forward deployment of the container. As before, it uses the graph configuration kept in project-config which is loaded in with grafyaml, which is included in the container. Once nice advantage is that it makes it quite easy to develop graphs locally, using the container which can talk to the public graphite instance. The documentation has been updated with a reference on how to do this. Change-Id: I0cc76d29b6911aecfebc71e5fdfe7cf4fcd071a4
2020-06-23 11:41:27 +10:00
- system-config-run-grafana:
dependencies:
- name: opendev-buildset-registry
Graphite container deployment This deploys graphite from the upstream container. We override the statsd configuration to have it listen on ipv6. Similarly we override the ngnix config to listen on ipv6, enable ssl, forward port 80 to 443, block the /admin page (we don't use it). For production we will just want to put some cinder storage in /opt/graphite/storage on the production host and figure out how to migrate the old stats. The is also a bit of cleanup that will follow, because we half-converted grafana01.opendev.org -- so everything can't be in the same group till that is gone. Testing has been added to push some stats and ensure they are seen. Change-Id: Ie843b3d90a72564ef90805f820c8abc61a71017d
2020-06-26 10:23:16 +10:00
- system-config-run-graphite
Add a keycloak server This adds a keycloak server so we can start experimenting with it. It's based on the docker-compose file Matthieu made for Zuul (see https://review.opendev.org/819745 ) We should be able to configure a realm and federate with openstackid and other providers as described in the opendev auth spec. However, I am unable to test federation with openstackid due its inability to configure an oauth app at "localhost". Therefore, we will need an actual deployed system to test it. This should allow us to do so. It will also allow use to connect realms to the newly available Zuul admin api on opendev. It should be possible to configure the realm the way we want, then export its configuration into a JSON file and then have our playbooks or the docker-compose file import it. That would allow us to drive change to the configuration of the system through code review. Because of the above limitation with openstackid, I think we should regard the current implementation as experimental. Once we have a realm configuration that we like (which we will create using the GUI), we can chose to either continue to maintain the config with the GUI and appropriate file backups, or switch to a gitops model based on an export. My understanding is that all the data (realms configuration and session) are kept in an H2 database. This is probably sufficient for now and even production use with Zuul, but we should probably switch to mariadb before any heavy (eg gerrit, etc) production use. This is a partial implementation of https://docs.opendev.org/opendev/infra-specs/latest/specs/central-auth.html We can re-deploy with a new domain when it exists. Change-Id: I2e069b1b220dbd3e0a5754ac094c2b296c141753 Co-Authored-By: Matthieu Huin <mhuin@redhat.com>
2021-11-30 13:03:12 -08:00
- system-config-run-keycloak
Add Gerrit 3.10 image builds and testing This adds Gerrit 3.10 image builds and our system-config-run-gerrit jobs for Gerrit 3.10. Change-Id: I6525090f78a1f1440f2cccda96f83158cb08d2ac
2024-05-31 14:41:54 -07:00
- system-config-run-review-3.10:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-gerrit-3.10
soft: true
Add Gerrit 3.11 image builds and testing This adds Gerrit 3.11 image builds an a system-config-run job for a 3.11 deployment. We don't add 3.10 to 3.11 upgrade testing yet. This will happen as a followup once we're happy with general image building and testing. This adds explicit push and force push permissions to the Administrators group via code reviewed changes. The reason for this is Gerrit 3.11 has decided that everything should be code reviewed by default including actions by Administrators. Since we're just doing this for bootstrapping of testing we quickly add the old expected permissions back via code review. This has a side effect of bumping up the change numbers for the cahnges we care about in review up by 2 integer values. The tests and screenshots are all updated to accomodate the shift in change numbers. Change-Id: Ib7090563a869577ed100ca71c496e8779e5b1010
2024-12-06 11:42:53 -08:00
- system-config-run-review-3.11:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-gerrit-3.11
soft: true
Drop Gerrit 3.9 image builds and testing Once we're satisfied that we will not be downgrading from Gerrit 3.10 to 3.9 we can safely drop the 3.9 image builds and testing. Followups will add forward looking 3.11 image builds and testing. Depends-On: https://review.opendev.org/c/openstack/project-config/+/937268 Change-Id: Ie46cd8b74147c835fbe5e414d13b3f7d3237c7cf
2024-12-06 11:35:04 -08:00
# Disabled until we can test the 3.10 -> 3.11 upgrade
#- system-config-upgrade-review:
# dependencies:
# - name: opendev-buildset-registry
# - name: system-config-build-image-gerrit-3.9
# soft: true
# - name: system-config-build-image-gerrit-3.10
# soft: true
Deploy refstack with ansible docker This adds a dockerfile to build an opendevorg/refstack image as well as the jobs to build and publish it. Change-Id: Icade6c713fa9bf6ab508fd4d8d65debada2ddb30
2020-01-31 09:27:34 -08:00
- system-config-build-image-refstack
- system-config-run-refstack:
dependencies:
Cleanup refstack job dependencies We need to depend on the buildset registry as we are building this image in a separate job. We also don't need to depend on the build job in gate, we only need the upload job. Change-Id: Ie7c2ed29c028f8c23d67ad38edbe04b12e22d026
2021-02-10 15:11:54 -08:00
- name: opendev-buildset-registry
Deploy refstack with ansible docker This adds a dockerfile to build an opendevorg/refstack image as well as the jobs to build and publish it. Change-Id: Icade6c713fa9bf6ab508fd4d8d65debada2ddb30
2020-01-31 09:27:34 -08:00
- name: system-config-build-image-refstack
soft: true
Add Jaeger tracing server Change-Id: I1aa68b1d5f99364fa09776301894b922ed169a3a
2022-09-05 13:42:18 -07:00
- system-config-run-tracing
Add zookeeper-statsd This adds a program, zookeeper-statsd, which monitors zookeeper metrics and reports them to statsd. It also adds a container to run that program. And it runs the container on each of the ZooKeeper quorum members. And it updates the graphite host to allow statsd traffic from quorum members. And it updates the 4-letter-word whitelist to allow the mntr command (which is used to gather metrics) to be issued. Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 12:57:21 -07:00
- system-config-run-zookeeper:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-zookeeper-statsd
soft: true
- system-config-run-zuul:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-zookeeper-statsd
soft: true
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-run-zuul-preview
- system-config-run-letsencrypt
Add assets and a related docker image/bundle Move some common assets into a top-level assets/ directory. Services can reference these assets via https://opendev.org/opendev/system-config/raw/branch/master/assets/<file> in <img> tags, etc. Some services want to embed these into their images, but we wish to only keep one canonical copy. For this, add a Dockerfile and jobs that creates a simple bundle of assets in opendevorg/assets. This can be referenced in other builds; the new BuildKit bind-mount is particularly useful for this (c.f. I36bdc76471eec5380a676ebcdd885a88d3985976). Change-Id: I3931566eb86a0618705d276445fa0a5f659692ea
2021-08-25 12:54:28 +10:00
- system-config-build-image-assets
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-build-image-jinja-init:
dependencies:
- name: opendev-buildset-registry
Fix job dependencies on old container images We recently updated all of these items to build on newer container images but some of the job dependencies for that were missed. Fix this before we start cleaning up the old container builds. Change-Id: I90946007026f9ca84f4fc6f0e5fe5fbf76d21627
2023-10-16 13:50:39 -07:00
- name: system-config-build-image-python-base-3.11-bookworm
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
soft: true
- system-config-build-image-gitea-init:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-jinja-init
soft: true
Migrate codesearch site to container The hound project has undergone a small re-birth and moved to https://github.com/hound-search/hound which has broken our deployment. We've talked about leaving codesearch up to gitea, but it's not quite there yet. There seems to be no point working on the puppet now. This builds a container than runs houndd. It's an opendev specific container; the config is pulled from project-config directly. There's some custom scripts that drive things. Some points for reviewers: - update-hound-config.sh uses "create-hound-config" (which is in jeepyb for historical reasons) to generate the config file. It grabs the latest projects.yaml from project-config and exits with a return code to indicate if things changed. - when the container starts, it runs update-hound-config.sh to populate the initial config. There is a testing environment flag and small config so it doesn't have to clone the entire opendev for functional testing. - it runs under supervisord so we can restart the daemon when projects are updated. Unlike earlier versions that didn't start listening till indexing was done, this version now puts up a "Hound is not ready yet" message when while it is working; so we can drop all the magic we were doing to probe if hound is listening via netstat and making Apache redirect to a status page. - resync-hound.sh is run from an external cron job daily, and does this update and restart check. Since it only reloads if changes are made, this should be relatively rare anyway. - There is a PR to monitor the config file (https://github.com/hound-search/hound/pull/357) which would mean the restart is unnecessary. This would be good in the near and we could remove the cron job. - playbooks/roles/codesearch is unexciting and deploys the container, certificates and an apache proxy back to localhost:6080 where hound is listening. I've combined removal of the old puppet bits here as the "-codesearch" namespace was already being used. Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
2020-11-17 17:13:46 +11:00
- system-config-build-image-hound:
dependencies:
- name: opendev-buildset-registry
Fix job dependencies on old container images We recently updated all of these items to build on newer container images but some of the job dependencies for that were missed. Fix this before we start cleaning up the old container builds. Change-Id: I90946007026f9ca84f4fc6f0e5fe5fbf76d21627
2023-10-16 13:50:39 -07:00
- name: system-config-build-image-python-base-3.11-bookworm
Migrate codesearch site to container The hound project has undergone a small re-birth and moved to https://github.com/hound-search/hound which has broken our deployment. We've talked about leaving codesearch up to gitea, but it's not quite there yet. There seems to be no point working on the puppet now. This builds a container than runs houndd. It's an opendev specific container; the config is pulled from project-config directly. There's some custom scripts that drive things. Some points for reviewers: - update-hound-config.sh uses "create-hound-config" (which is in jeepyb for historical reasons) to generate the config file. It grabs the latest projects.yaml from project-config and exits with a return code to indicate if things changed. - when the container starts, it runs update-hound-config.sh to populate the initial config. There is a testing environment flag and small config so it doesn't have to clone the entire opendev for functional testing. - it runs under supervisord so we can restart the daemon when projects are updated. Unlike earlier versions that didn't start listening till indexing was done, this version now puts up a "Hound is not ready yet" message when while it is working; so we can drop all the magic we were doing to probe if hound is listening via netstat and making Apache redirect to a status page. - resync-hound.sh is run from an external cron job daily, and does this update and restart check. Since it only reloads if changes are made, this should be relatively rare anyway. - There is a PR to monitor the config file (https://github.com/hound-search/hound/pull/357) which would mean the restart is unnecessary. This would be good in the near and we could remove the cron job. - playbooks/roles/codesearch is unexciting and deploys the container, certificates and an apache proxy back to localhost:6080 where hound is listening. I've combined removal of the old puppet bits here as the "-codesearch" namespace was already being used. Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
2020-11-17 17:13:46 +11:00
soft: true
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-build-image-etherpad
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
- system-config-build-image-mailman
gitea: use assets bundle This uses the opendev assets bundle image created with I3166679bde6d771276289b9d32e7e4407957b2f8. The mount options require using BuildKit, hence the Dockerfile update. Otherwise conceptually it's fairly simple; copy in the files from the opendevorg/assets image rather than the file-system. Change-Id: I36bdc76471eec5380a676ebcdd885a88d3985976
2021-08-25 12:55:21 +10:00
- system-config-build-image-gitea:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-assets
soft: true
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-build-image-haproxy-statsd:
dependencies:
- name: opendev-buildset-registry
Convert haproxy-statsd to bookworm base image We keep the same python 3.11 version we had on bullseye but switch the rest of the userspace to bookworm. Since the python version doesn't change this should have minimal impact. Change-Id: I59ad8c2a92159f51d567dd0212e2ab8bec1b45b1
2023-09-14 13:36:18 -07:00
- name: system-config-build-image-python-base-3.11-bookworm
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
soft: true
Add zookeeper-statsd This adds a program, zookeeper-statsd, which monitors zookeeper metrics and reports them to statsd. It also adds a container to run that program. And it runs the container on each of the ZooKeeper quorum members. And it updates the graphite host to allow statsd traffic from quorum members. And it updates the 4-letter-word whitelist to allow the mntr command (which is used to gather metrics) to be issued. Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 12:57:21 -07:00
- system-config-build-image-zookeeper-statsd:
dependencies:
- name: opendev-buildset-registry
Fix job dependencies on old container images We recently updated all of these items to build on newer container images but some of the job dependencies for that were missed. Fix this before we start cleaning up the old container builds. Change-Id: I90946007026f9ca84f4fc6f0e5fe5fbf76d21627
2023-10-16 13:50:39 -07:00
- name: system-config-build-image-python-base-3.11-bookworm
Add zookeeper-statsd This adds a program, zookeeper-statsd, which monitors zookeeper metrics and reports them to statsd. It also adds a container to run that program. And it runs the container on each of the ZooKeeper quorum members. And it updates the graphite host to allow statsd traffic from quorum members. And it updates the 4-letter-word whitelist to allow the mntr command (which is used to gather metrics) to be issued. Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 12:57:21 -07:00
soft: true
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-build-image-accessbot:
dependencies:
- name: opendev-buildset-registry
Update accessbot,ircboot and matrix-eavesdrop to bookworm container Change-Id: Idb096e21f57df3db1acc3d3afe3881e2e540f965
2023-07-11 13:28:06 +10:00
- name: system-config-build-image-python-base-3.11-bookworm
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
soft: true
Create ircbot container This container installs Limnoria, the supybot replacement as the generic ircbot container. We install meetbot plugin as a sibling project. Previously we've conflated supybot with meetbot, which is a bit confusing because meetbot is a plugin, but we also use other plugins such as the channel logger. We also hope to convert some of our other bots to Limnoria (ptgbot?) to consolidate everything. For this reason I've called this the more generic "ircbot". The image installs meetbot as a sibling project, with the idea being any other plugins would also be installed as siblings. The siblings install expects the work directory to be a relative directory. I'm not sure we run this from other projects, but this will work the same if we do. Depends-On: https://review.opendev.org/c/opendev/meetbot/+/793876 Change-Id: Icee4c6bbb5ea235ba69c10f800a14bbf5beef3d5
2021-05-31 12:32:41 +10:00
- system-config-build-image-ircbot:
dependencies:
- name: opendev-buildset-registry
Fix job dependencies on old container images We recently updated all of these items to build on newer container images but some of the job dependencies for that were missed. Fix this before we start cleaning up the old container builds. Change-Id: I90946007026f9ca84f4fc6f0e5fe5fbf76d21627
2023-10-16 13:50:39 -07:00
- name: system-config-build-image-python-builder-3.11-bookworm
Create ircbot container This container installs Limnoria, the supybot replacement as the generic ircbot container. We install meetbot plugin as a sibling project. Previously we've conflated supybot with meetbot, which is a bit confusing because meetbot is a plugin, but we also use other plugins such as the channel logger. We also hope to convert some of our other bots to Limnoria (ptgbot?) to consolidate everything. For this reason I've called this the more generic "ircbot". The image installs meetbot as a sibling project, with the idea being any other plugins would also be installed as siblings. The siblings install expects the work directory to be a relative directory. I'm not sure we run this from other projects, but this will work the same if we do. Depends-On: https://review.opendev.org/c/opendev/meetbot/+/793876 Change-Id: Icee4c6bbb5ea235ba69c10f800a14bbf5beef3d5
2021-05-31 12:32:41 +10:00
soft: true
Add matrix-eavesdrop container image This builds a container image with a simple eavesdrop bot for Matrix. Change-Id: I5304b4ec974b84886ac969b59cfcec8dec2febf9
2021-07-09 17:13:51 -07:00
- system-config-build-image-matrix-eavesdrop:
dependencies:
- name: opendev-buildset-registry
Fix job dependencies on old container images We recently updated all of these items to build on newer container images but some of the job dependencies for that were missed. Fix this before we start cleaning up the old container builds. Change-Id: I90946007026f9ca84f4fc6f0e5fe5fbf76d21627
2023-10-16 13:50:39 -07:00
- name: system-config-build-image-python-builder-3.11-bookworm
Produce both buster and bullseye container images This will allow us to roll out consumption forward in a methodical manner. This reverts commit 45caec4d43900bc66fb0b8c219c6dcc3180ca8aa. Note the weird ARG definitions are there for a reason: https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact Change-Id: I81174ac035164695f1c27c9662f25335b78c2e64
2021-08-27 13:52:25 -05:00
soft: true
Add Debian bookworm based python images uWSGI: libffi7 has been retired with bullseye so update to libffi8 Change-Id: I3e7e05b0e0148d73e7f0087f456a3f8de10e0c87
2023-07-11 12:32:18 +10:00
- system-config-build-image-python-base-3.10-bookworm
- system-config-build-image-python-base-3.11-bookworm
Add a debug python base image This adds a python-base:3.11-bookworm-debug image, which is built on the normal python:3.11-bookworm upstream image instead of the slim upstream image. The normal image includes debug symbols for the python interpreter which is compiled during its build phase, so this is the best way to get an opendev python-base image with debug symbols. Change-Id: I1d89ac947cd3bea8a468f3ee022fb4cc93bece1f
2023-10-10 08:03:31 -07:00
- system-config-build-image-python-base-3.11-bookworm-debug
Add python3.12 bookworm base images This is probably of limited utility currently as we don't have any good test platforms for running jobs on top of python3.12 outside of these container images. That said this will ensure our base job setup works with python3.12, and give early adopters the option of working with python3.12 in this way. Change-Id: Ief76c76e86c8cbae50e9fc7cb34356fa1c8e99a8
2023-10-18 13:38:14 -07:00
- system-config-build-image-python-base-3.12-bookworm
- system-config-build-image-python-base-3.12-bookworm-debug
Add Debian bookworm based python images uWSGI: libffi7 has been retired with bullseye so update to libffi8 Change-Id: I3e7e05b0e0148d73e7f0087f456a3f8de10e0c87
2023-07-11 12:32:18 +10:00
- system-config-build-image-python-builder-3.10-bookworm
- system-config-build-image-python-builder-3.11-bookworm
Add python3.12 bookworm base images This is probably of limited utility currently as we don't have any good test platforms for running jobs on top of python3.12 outside of these container images. That said this will ensure our base job setup works with python3.12, and give early adopters the option of working with python3.12 in this way. Change-Id: Ief76c76e86c8cbae50e9fc7cb34356fa1c8e99a8
2023-10-18 13:38:14 -07:00
- system-config-build-image-python-builder-3.12-bookworm
Add Debian bookworm based python images uWSGI: libffi7 has been retired with bullseye so update to libffi8 Change-Id: I3e7e05b0e0148d73e7f0087f456a3f8de10e0c87
2023-07-11 12:32:18 +10:00
- system-config-build-image-uwsgi-base-3.10-bookworm
- system-config-build-image-uwsgi-base-3.11-bookworm
Add python3.12 bookworm base images This is probably of limited utility currently as we don't have any good test platforms for running jobs on top of python3.12 outside of these container images. That said this will ensure our base job setup works with python3.12, and give early adopters the option of working with python3.12 in this way. Change-Id: Ief76c76e86c8cbae50e9fc7cb34356fa1c8e99a8
2023-10-18 13:38:14 -07:00
- system-config-build-image-uwsgi-base-3.12-bookworm
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
check-arm64:
jobs:
- system-config-run-base-arm64
- system-config-run-mirror-arm64
gate:
jobs:
- opendev-tox-docs
Run opendev-buildset-regitry on Noble This unpins the nodeset for opendev-buildset-registry which should start working once the parent change has landed and our new mirror content has been copied and deployed. We'll need to recheck this change when we believe we've reached that state. Change-Id: I7a92036cc63963d3066b4eb2bf451dd0ec8a887b
2024-08-21 16:43:16 -07:00
- opendev-buildset-registry
Set tox-linters job timeout to the default This job only takes a few minutes to run. There isn't any reason to use an hour long timeout on this job as a result. Reset it back to the default and simplify the config. Change-Id: Id808e2fa35caa7cf627014b04d1560326eeebc5b
2024-08-22 13:10:26 -07:00
- tox-linters
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-run-base
system-config-run-borg-backup: add to gate We must have missed this, I noticed when it didn't run on the gate job for I949c40e9046008d4f442b322a267ce0c967a99dc Change-Id: I62c5c0f262d9bd53580367dc9f1ad00fe7b6f6f2
2022-08-11 13:54:52 +10:00
- system-config-run-borg-backup
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-run-dns
- system-config-run-eavesdrop:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-accessbot
soft: true
limnoria/meetbot setup on eavesdrop01.opendev.org This installs our Limnoira/meetbot container and configures it on eavesdrop01.opendev.org. I have ported the configuration from the old puppet as best I can (it is very verbose); my procedure was to use the Limnoira wizard to start a new config file then backport everything from the old file. I felt this was best to not miss any new options. This does channel logging (via built-in ChannelLogger plugin, along with a cron job for logs2html) and runs our fork of meetbot. It exports the channel logs via HTTP to /irclogs and meetings logs to /meetings. meetings.opendev.org will proxy to these two locations when the server is active. Note this has not ported the channel list; so the bot will not be listening in our channels. Change-Id: I9f9a466c271e1a706f9f98f816de0e84047519f1
2021-05-31 14:47:07 +10:00
- name: system-config-upload-image-ircbot
soft: true
Run matrix-eavesdrop on eavesdrop Thin runs the new matrix-eavesdrop bot on the eavesdrop server. It will write logs out to the limnoria logs directory, which is mounted inside the container. Change-Id: I867eec692f63099b295a37a028ee096c24109a2e
2021-07-09 18:39:59 -07:00
- name: system-config-upload-image-matrix-eavesdrop
soft: true
Migrate codesearch site to container The hound project has undergone a small re-birth and moved to https://github.com/hound-search/hound which has broken our deployment. We've talked about leaving codesearch up to gitea, but it's not quite there yet. There seems to be no point working on the puppet now. This builds a container than runs houndd. It's an opendev specific container; the config is pulled from project-config directly. There's some custom scripts that drive things. Some points for reviewers: - update-hound-config.sh uses "create-hound-config" (which is in jeepyb for historical reasons) to generate the config file. It grabs the latest projects.yaml from project-config and exits with a return code to indicate if things changed. - when the container starts, it runs update-hound-config.sh to populate the initial config. There is a testing environment flag and small config so it doesn't have to clone the entire opendev for functional testing. - it runs under supervisord so we can restart the daemon when projects are updated. Unlike earlier versions that didn't start listening till indexing was done, this version now puts up a "Hound is not ready yet" message when while it is working; so we can drop all the magic we were doing to probe if hound is listening via netstat and making Apache redirect to a status page. - resync-hound.sh is run from an external cron job daily, and does this update and restart check. Since it only reloads if changes are made, this should be relatively rare anyway. - There is a PR to monitor the config file (https://github.com/hound-search/hound/pull/357) which would mean the restart is unnecessary. This would be good in the near and we could remove the cron job. - playbooks/roles/codesearch is unexciting and deploys the container, certificates and an apache proxy back to localhost:6080 where hound is listening. I've combined removal of the old puppet bits here as the "-codesearch" namespace was already being used. Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
2020-11-17 17:13:46 +11:00
- system-config-run-codesearch:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-hound
soft: true
kerberos-kdc: role to manage Kerberos KDC servers This adds a role and related testing to manage our Kerberos KDC servers, intended to replace the puppet modules currently performing this task. This role automates realm creation, initial setup, key material distribution and replica host configuration. None of this is intended to run on the production servers which are already setup with an active database, and the role should be effectively idempotent in production. Note that this does not yet switch the production servers into the new groups; this can be done in a separate step under controlled conditions and with related upgrades of the host OS to Focal. Change-Id: I60b40897486b29beafc76025790c501b5055313d
2021-03-05 16:10:01 +11:00
- system-config-run-kerberos
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
- system-config-run-lists3:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-mailman
soft: true
Add zookeeper-statsd This adds a program, zookeeper-statsd, which monitors zookeeper metrics and reports them to statsd. It also adds a container to run that program. And it runs the container on each of the ZooKeeper quorum members. And it updates the graphite host to allow statsd traffic from quorum members. And it updates the 4-letter-word whitelist to allow the mntr command (which is used to gather metrics) to be issued. Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 12:57:21 -07:00
- system-config-run-nodepool:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-zookeeper-statsd
soft: true
Use upstream jitsi-meet web image This has our change to open etherpad on join, so we should no longer need to run a fork of the web server. Switch to the upstream container image and stop building our own. Change-Id: I3e8da211c78b6486a3dcbd362ae7eb03cc9f5a48
2021-03-02 12:59:37 -08:00
- system-config-run-meetpad
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-run-mirror-x86
- system-config-run-mirror-update
Add paste service The paste service needs an upgrade; since others have created a lodgeit container it seems worth us keeping the service going if only to maintain the historical corpus of pastes. This adds the ansible to deploy lodgeit and a sibling mariadb container. I have imported a dump of the old data as a test. The dump is ~4gb and imported it takes up about double that; certainly nothing we need to be too concerned over. The server will be more than capable of running the db container alongside the lodgeit instance. This should have no effect on production until we decide to switch DNS. Change-Id: I284864217aa49d664ddc3ebdc800383b2d7e00e3
2021-06-29 11:57:10 +10:00
- system-config-run-paste:
dependencies:
- name: opendev-buildset-registry
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-run-static
- system-config-run-docker-registry
- system-config-run-etherpad:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-etherpad
soft: true
- system-config-run-gitea:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-gitea
soft: true
- name: system-config-upload-image-haproxy-statsd
soft: true
Grafana container deployment This uses the Grafana container created with Iddfafe852166fe95b3e433420e2e2a4a6380fc64 to run the grafana.opendev.org service. We retain the old model of an Apache reverse-proxy; it's well tested and understood, it's much easier than trying to map all the SSL termination/renewal/etc. into the Grafana container and we don't have to convince ourselves the container is safe to be directly web-facing. Otherwise this is a fairly straight forward deployment of the container. As before, it uses the graph configuration kept in project-config which is loaded in with grafyaml, which is included in the container. Once nice advantage is that it makes it quite easy to develop graphs locally, using the container which can talk to the public graphite instance. The documentation has been updated with a reference on how to do this. Change-Id: I0cc76d29b6911aecfebc71e5fdfe7cf4fcd071a4
2020-06-23 11:41:27 +10:00
- system-config-run-grafana:
dependencies:
- name: opendev-buildset-registry
Graphite container deployment This deploys graphite from the upstream container. We override the statsd configuration to have it listen on ipv6. Similarly we override the ngnix config to listen on ipv6, enable ssl, forward port 80 to 443, block the /admin page (we don't use it). For production we will just want to put some cinder storage in /opt/graphite/storage on the production host and figure out how to migrate the old stats. The is also a bit of cleanup that will follow, because we half-converted grafana01.opendev.org -- so everything can't be in the same group till that is gone. Testing has been added to push some stats and ensure they are seen. Change-Id: Ie843b3d90a72564ef90805f820c8abc61a71017d
2020-06-26 10:23:16 +10:00
- system-config-run-graphite
Add a keycloak server This adds a keycloak server so we can start experimenting with it. It's based on the docker-compose file Matthieu made for Zuul (see https://review.opendev.org/819745 ) We should be able to configure a realm and federate with openstackid and other providers as described in the opendev auth spec. However, I am unable to test federation with openstackid due its inability to configure an oauth app at "localhost". Therefore, we will need an actual deployed system to test it. This should allow us to do so. It will also allow use to connect realms to the newly available Zuul admin api on opendev. It should be possible to configure the realm the way we want, then export its configuration into a JSON file and then have our playbooks or the docker-compose file import it. That would allow us to drive change to the configuration of the system through code review. Because of the above limitation with openstackid, I think we should regard the current implementation as experimental. Once we have a realm configuration that we like (which we will create using the GUI), we can chose to either continue to maintain the config with the GUI and appropriate file backups, or switch to a gitops model based on an export. My understanding is that all the data (realms configuration and session) are kept in an H2 database. This is probably sufficient for now and even production use with Zuul, but we should probably switch to mariadb before any heavy (eg gerrit, etc) production use. This is a partial implementation of https://docs.opendev.org/opendev/infra-specs/latest/specs/central-auth.html We can re-deploy with a new domain when it exists. Change-Id: I2e069b1b220dbd3e0a5754ac094c2b296c141753 Co-Authored-By: Matthieu Huin <mhuin@redhat.com>
2021-11-30 13:03:12 -08:00
- system-config-run-keycloak
Add Gerrit 3.10 image builds and testing This adds Gerrit 3.10 image builds and our system-config-run-gerrit jobs for Gerrit 3.10. Change-Id: I6525090f78a1f1440f2cccda96f83158cb08d2ac
2024-05-31 14:41:54 -07:00
- system-config-run-review-3.10:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-gerrit-3.10
soft: true
Add Gerrit 3.11 image builds and testing This adds Gerrit 3.11 image builds an a system-config-run job for a 3.11 deployment. We don't add 3.10 to 3.11 upgrade testing yet. This will happen as a followup once we're happy with general image building and testing. This adds explicit push and force push permissions to the Administrators group via code reviewed changes. The reason for this is Gerrit 3.11 has decided that everything should be code reviewed by default including actions by Administrators. Since we're just doing this for bootstrapping of testing we quickly add the old expected permissions back via code review. This has a side effect of bumping up the change numbers for the cahnges we care about in review up by 2 integer values. The tests and screenshots are all updated to accomodate the shift in change numbers. Change-Id: Ib7090563a869577ed100ca71c496e8779e5b1010
2024-12-06 11:42:53 -08:00
- system-config-run-review-3.11:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-gerrit-3.11
soft: true
refstack: add production image and deployment jobs Change-Id: I017a32ee374f0473525c9941c41b26c2a43bf2c8
2021-02-09 16:15:08 +11:00
- system-config-run-refstack:
dependencies:
Cleanup refstack job dependencies We need to depend on the buildset registry as we are building this image in a separate job. We also don't need to depend on the build job in gate, we only need the upload job. Change-Id: Ie7c2ed29c028f8c23d67ad38edbe04b12e22d026
2021-02-10 15:11:54 -08:00
- name: opendev-buildset-registry
refstack: add production image and deployment jobs Change-Id: I017a32ee374f0473525c9941c41b26c2a43bf2c8
2021-02-09 16:15:08 +11:00
- name: system-config-upload-image-refstack
soft: true
Add Jaeger tracing server Change-Id: I1aa68b1d5f99364fa09776301894b922ed169a3a
2022-09-05 13:42:18 -07:00
- system-config-run-tracing
Add zookeeper-statsd This adds a program, zookeeper-statsd, which monitors zookeeper metrics and reports them to statsd. It also adds a container to run that program. And it runs the container on each of the ZooKeeper quorum members. And it updates the graphite host to allow statsd traffic from quorum members. And it updates the 4-letter-word whitelist to allow the mntr command (which is used to gather metrics) to be issued. Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 12:57:21 -07:00
- system-config-run-zookeeper:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-zookeeper-statsd
soft: true
- system-config-run-zuul:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-zookeeper-statsd
soft: true
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-run-zuul-preview
- system-config-run-letsencrypt
- system-config-upload-image-jinja-init:
dependencies:
- name: opendev-buildset-registry
Fix job dependencies on old container images We recently updated all of these items to build on newer container images but some of the job dependencies for that were missed. Fix this before we start cleaning up the old container builds. Change-Id: I90946007026f9ca84f4fc6f0e5fe5fbf76d21627
2023-10-16 13:50:39 -07:00
- name: system-config-upload-image-python-base-3.11-bookworm
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
soft: true
- system-config-upload-image-gitea-init:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-jinja-init
soft: true
Update Hound to Use Python 3.11 Base Images This commits update hound Dockerfile to use Python 3.11 images instead of older 3.9 one. It's part of the effort to update Python version to 3.10 or later. Change-Id: Idf5076709c55d3327c69c6e8160a8348b6640ca7
2023-04-20 15:26:28 +09:00
- system-config-upload-image-hound:
dependencies:
- name: opendev-buildset-registry
Fix job dependencies on old container images We recently updated all of these items to build on newer container images but some of the job dependencies for that were missed. Fix this before we start cleaning up the old container builds. Change-Id: I90946007026f9ca84f4fc6f0e5fe5fbf76d21627
2023-10-16 13:50:39 -07:00
- name: system-config-build-image-python-base-3.11-bookworm
Update Hound to Use Python 3.11 Base Images This commits update hound Dockerfile to use Python 3.11 images instead of older 3.9 one. It's part of the effort to update Python version to 3.10 or later. Change-Id: Idf5076709c55d3327c69c6e8160a8348b6640ca7
2023-04-20 15:26:28 +09:00
soft: true
Add assets and a related docker image/bundle Move some common assets into a top-level assets/ directory. Services can reference these assets via https://opendev.org/opendev/system-config/raw/branch/master/assets/<file> in <img> tags, etc. Some services want to embed these into their images, but we wish to only keep one canonical copy. For this, add a Dockerfile and jobs that creates a simple bundle of assets in opendevorg/assets. This can be referenced in other builds; the new BuildKit bind-mount is particularly useful for this (c.f. I36bdc76471eec5380a676ebcdd885a88d3985976). Change-Id: I3931566eb86a0618705d276445fa0a5f659692ea
2021-08-25 12:54:28 +10:00
- system-config-upload-image-assets
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-upload-image-etherpad
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
- system-config-upload-image-mailman
gitea: use assets bundle This uses the opendev assets bundle image created with I3166679bde6d771276289b9d32e7e4407957b2f8. The mount options require using BuildKit, hence the Dockerfile update. Otherwise conceptually it's fairly simple; copy in the files from the opendevorg/assets image rather than the file-system. Change-Id: I36bdc76471eec5380a676ebcdd885a88d3985976
2021-08-25 12:55:21 +10:00
- system-config-upload-image-gitea:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-assets
soft: true
refstack: add production image and deployment jobs Change-Id: I017a32ee374f0473525c9941c41b26c2a43bf2c8
2021-02-09 16:15:08 +11:00
- system-config-upload-image-refstack
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-upload-image-haproxy-statsd:
dependencies:
- name: opendev-buildset-registry
Convert haproxy-statsd to bookworm base image We keep the same python 3.11 version we had on bullseye but switch the rest of the userspace to bookworm. Since the python version doesn't change this should have minimal impact. Change-Id: I59ad8c2a92159f51d567dd0212e2ab8bec1b45b1
2023-09-14 13:36:18 -07:00
- name: system-config-upload-image-python-base-3.11-bookworm
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
soft: true
Add zookeeper-statsd This adds a program, zookeeper-statsd, which monitors zookeeper metrics and reports them to statsd. It also adds a container to run that program. And it runs the container on each of the ZooKeeper quorum members. And it updates the graphite host to allow statsd traffic from quorum members. And it updates the 4-letter-word whitelist to allow the mntr command (which is used to gather metrics) to be issued. Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 12:57:21 -07:00
- system-config-upload-image-zookeeper-statsd:
dependencies:
- name: opendev-buildset-registry
Fix job dependencies on old container images We recently updated all of these items to build on newer container images but some of the job dependencies for that were missed. Fix this before we start cleaning up the old container builds. Change-Id: I90946007026f9ca84f4fc6f0e5fe5fbf76d21627
2023-10-16 13:50:39 -07:00
- name: system-config-upload-image-python-base-3.11-bookworm
Add zookeeper-statsd This adds a program, zookeeper-statsd, which monitors zookeeper metrics and reports them to statsd. It also adds a container to run that program. And it runs the container on each of the ZooKeeper quorum members. And it updates the graphite host to allow statsd traffic from quorum members. And it updates the 4-letter-word whitelist to allow the mntr command (which is used to gather metrics) to be issued. Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 12:57:21 -07:00
soft: true
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-upload-image-accessbot:
dependencies:
- name: opendev-buildset-registry
Update accessbot,ircboot and matrix-eavesdrop to bookworm container Change-Id: Idb096e21f57df3db1acc3d3afe3881e2e540f965
2023-07-11 13:28:06 +10:00
- name: system-config-upload-image-python-base-3.11-bookworm
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
soft: true
Create ircbot container This container installs Limnoria, the supybot replacement as the generic ircbot container. We install meetbot plugin as a sibling project. Previously we've conflated supybot with meetbot, which is a bit confusing because meetbot is a plugin, but we also use other plugins such as the channel logger. We also hope to convert some of our other bots to Limnoria (ptgbot?) to consolidate everything. For this reason I've called this the more generic "ircbot". The image installs meetbot as a sibling project, with the idea being any other plugins would also be installed as siblings. The siblings install expects the work directory to be a relative directory. I'm not sure we run this from other projects, but this will work the same if we do. Depends-On: https://review.opendev.org/c/opendev/meetbot/+/793876 Change-Id: Icee4c6bbb5ea235ba69c10f800a14bbf5beef3d5
2021-05-31 12:32:41 +10:00
- system-config-upload-image-ircbot:
dependencies:
- name: opendev-buildset-registry
Fix job dependencies on old container images We recently updated all of these items to build on newer container images but some of the job dependencies for that were missed. Fix this before we start cleaning up the old container builds. Change-Id: I90946007026f9ca84f4fc6f0e5fe5fbf76d21627
2023-10-16 13:50:39 -07:00
- name: system-config-upload-image-python-builder-3.11-bookworm
Create ircbot container This container installs Limnoria, the supybot replacement as the generic ircbot container. We install meetbot plugin as a sibling project. Previously we've conflated supybot with meetbot, which is a bit confusing because meetbot is a plugin, but we also use other plugins such as the channel logger. We also hope to convert some of our other bots to Limnoria (ptgbot?) to consolidate everything. For this reason I've called this the more generic "ircbot". The image installs meetbot as a sibling project, with the idea being any other plugins would also be installed as siblings. The siblings install expects the work directory to be a relative directory. I'm not sure we run this from other projects, but this will work the same if we do. Depends-On: https://review.opendev.org/c/opendev/meetbot/+/793876 Change-Id: Icee4c6bbb5ea235ba69c10f800a14bbf5beef3d5
2021-05-31 12:32:41 +10:00
soft: true
Run matrix-eavesdrop on eavesdrop Thin runs the new matrix-eavesdrop bot on the eavesdrop server. It will write logs out to the limnoria logs directory, which is mounted inside the container. Change-Id: I867eec692f63099b295a37a028ee096c24109a2e
2021-07-09 18:39:59 -07:00
- system-config-upload-image-matrix-eavesdrop:
Add matrix-eavesdrop container image This builds a container image with a simple eavesdrop bot for Matrix. Change-Id: I5304b4ec974b84886ac969b59cfcec8dec2febf9
2021-07-09 17:13:51 -07:00
dependencies:
- name: opendev-buildset-registry
Fix job dependencies on old container images We recently updated all of these items to build on newer container images but some of the job dependencies for that were missed. Fix this before we start cleaning up the old container builds. Change-Id: I90946007026f9ca84f4fc6f0e5fe5fbf76d21627
2023-10-16 13:50:39 -07:00
- name: system-config-upload-image-python-builder-3.11-bookworm
Produce both buster and bullseye container images This will allow us to roll out consumption forward in a methodical manner. This reverts commit 45caec4d43900bc66fb0b8c219c6dcc3180ca8aa. Note the weird ARG definitions are there for a reason: https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact Change-Id: I81174ac035164695f1c27c9662f25335b78c2e64
2021-08-27 13:52:25 -05:00
soft: true
Add Debian bookworm based python images uWSGI: libffi7 has been retired with bullseye so update to libffi8 Change-Id: I3e7e05b0e0148d73e7f0087f456a3f8de10e0c87
2023-07-11 12:32:18 +10:00
- system-config-upload-image-python-base-3.10-bookworm
- system-config-upload-image-python-base-3.11-bookworm
Add a debug python base image This adds a python-base:3.11-bookworm-debug image, which is built on the normal python:3.11-bookworm upstream image instead of the slim upstream image. The normal image includes debug symbols for the python interpreter which is compiled during its build phase, so this is the best way to get an opendev python-base image with debug symbols. Change-Id: I1d89ac947cd3bea8a468f3ee022fb4cc93bece1f
2023-10-10 08:03:31 -07:00
- system-config-upload-image-python-base-3.11-bookworm-debug
Add python3.12 bookworm base images This is probably of limited utility currently as we don't have any good test platforms for running jobs on top of python3.12 outside of these container images. That said this will ensure our base job setup works with python3.12, and give early adopters the option of working with python3.12 in this way. Change-Id: Ief76c76e86c8cbae50e9fc7cb34356fa1c8e99a8
2023-10-18 13:38:14 -07:00
- system-config-upload-image-python-base-3.12-bookworm
- system-config-upload-image-python-base-3.12-bookworm-debug
Add Debian bookworm based python images uWSGI: libffi7 has been retired with bullseye so update to libffi8 Change-Id: I3e7e05b0e0148d73e7f0087f456a3f8de10e0c87
2023-07-11 12:32:18 +10:00
- system-config-upload-image-python-builder-3.10-bookworm
- system-config-upload-image-python-builder-3.11-bookworm
Add python3.12 bookworm base images This is probably of limited utility currently as we don't have any good test platforms for running jobs on top of python3.12 outside of these container images. That said this will ensure our base job setup works with python3.12, and give early adopters the option of working with python3.12 in this way. Change-Id: Ief76c76e86c8cbae50e9fc7cb34356fa1c8e99a8
2023-10-18 13:38:14 -07:00
- system-config-upload-image-python-builder-3.12-bookworm
Add Debian bookworm based python images uWSGI: libffi7 has been retired with bullseye so update to libffi8 Change-Id: I3e7e05b0e0148d73e7f0087f456a3f8de10e0c87
2023-07-11 12:32:18 +10:00
- system-config-upload-image-uwsgi-base-3.10-bookworm
- system-config-upload-image-uwsgi-base-3.11-bookworm
Add python3.12 bookworm base images This is probably of limited utility currently as we don't have any good test platforms for running jobs on top of python3.12 outside of these container images. That said this will ensure our base job setup works with python3.12, and give early adopters the option of working with python3.12 in this way. Change-Id: Ief76c76e86c8cbae50e9fc7cb34356fa1c8e99a8
2023-10-18 13:38:14 -07:00
- system-config-upload-image-uwsgi-base-3.12-bookworm
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
promote:
jobs:
- opendev-promote-docs
deploy:
jobs:
Add assets and a related docker image/bundle Move some common assets into a top-level assets/ directory. Services can reference these assets via https://opendev.org/opendev/system-config/raw/branch/master/assets/<file> in <img> tags, etc. Some services want to embed these into their images, but we wish to only keep one canonical copy. For this, add a Dockerfile and jobs that creates a simple bundle of assets in opendevorg/assets. This can be referenced in other builds; the new BuildKit bind-mount is particularly useful for this (c.f. I36bdc76471eec5380a676ebcdd885a88d3985976). Change-Id: I3931566eb86a0618705d276445fa0a5f659692ea
2021-08-25 12:54:28 +10:00
- system-config-promote-image-assets
Migrate codesearch site to container The hound project has undergone a small re-birth and moved to https://github.com/hound-search/hound which has broken our deployment. We've talked about leaving codesearch up to gitea, but it's not quite there yet. There seems to be no point working on the puppet now. This builds a container than runs houndd. It's an opendev specific container; the config is pulled from project-config directly. There's some custom scripts that drive things. Some points for reviewers: - update-hound-config.sh uses "create-hound-config" (which is in jeepyb for historical reasons) to generate the config file. It grabs the latest projects.yaml from project-config and exits with a return code to indicate if things changed. - when the container starts, it runs update-hound-config.sh to populate the initial config. There is a testing environment flag and small config so it doesn't have to clone the entire opendev for functional testing. - it runs under supervisord so we can restart the daemon when projects are updated. Unlike earlier versions that didn't start listening till indexing was done, this version now puts up a "Hound is not ready yet" message when while it is working; so we can drop all the magic we were doing to probe if hound is listening via netstat and making Apache redirect to a status page. - resync-hound.sh is run from an external cron job daily, and does this update and restart check. Since it only reloads if changes are made, this should be relatively rare anyway. - There is a PR to monitor the config file (https://github.com/hound-search/hound/pull/357) which would mean the restart is unnecessary. This would be good in the near and we could remove the cron job. - playbooks/roles/codesearch is unexciting and deploys the container, certificates and an apache proxy back to localhost:6080 where hound is listening. I've combined removal of the old puppet bits here as the "-codesearch" namespace was already being used. Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
2020-11-17 17:13:46 +11:00
- system-config-promote-image-hound
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-promote-image-jinja-init
- system-config-promote-image-gitea-init
- system-config-promote-image-gitea
- system-config-promote-image-etherpad
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
- system-config-promote-image-mailman
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-promote-image-haproxy-statsd
Add zookeeper-statsd This adds a program, zookeeper-statsd, which monitors zookeeper metrics and reports them to statsd. It also adds a container to run that program. And it runs the container on each of the ZooKeeper quorum members. And it updates the graphite host to allow statsd traffic from quorum members. And it updates the 4-letter-word whitelist to allow the mntr command (which is used to gather metrics) to be issued. Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 12:57:21 -07:00
- system-config-promote-image-zookeeper-statsd
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- system-config-promote-image-accessbot
refstack: add production image and deployment jobs Change-Id: I017a32ee374f0473525c9941c41b26c2a43bf2c8
2021-02-09 16:15:08 +11:00
- system-config-promote-image-refstack
Create ircbot container This container installs Limnoria, the supybot replacement as the generic ircbot container. We install meetbot plugin as a sibling project. Previously we've conflated supybot with meetbot, which is a bit confusing because meetbot is a plugin, but we also use other plugins such as the channel logger. We also hope to convert some of our other bots to Limnoria (ptgbot?) to consolidate everything. For this reason I've called this the more generic "ircbot". The image installs meetbot as a sibling project, with the idea being any other plugins would also be installed as siblings. The siblings install expects the work directory to be a relative directory. I'm not sure we run this from other projects, but this will work the same if we do. Depends-On: https://review.opendev.org/c/opendev/meetbot/+/793876 Change-Id: Icee4c6bbb5ea235ba69c10f800a14bbf5beef3d5
2021-05-31 12:32:41 +10:00
- system-config-promote-image-ircbot
Add matrix-eavesdrop container image This builds a container image with a simple eavesdrop bot for Matrix. Change-Id: I5304b4ec974b84886ac969b59cfcec8dec2febf9
2021-07-09 17:13:51 -07:00
- system-config-promote-image-matrix-eavesdrop
Add Debian bookworm based python images uWSGI: libffi7 has been retired with bullseye so update to libffi8 Change-Id: I3e7e05b0e0148d73e7f0087f456a3f8de10e0c87
2023-07-11 12:32:18 +10:00
- system-config-promote-image-python-base-3.10-bookworm
- system-config-promote-image-python-base-3.11-bookworm
Add a debug python base image This adds a python-base:3.11-bookworm-debug image, which is built on the normal python:3.11-bookworm upstream image instead of the slim upstream image. The normal image includes debug symbols for the python interpreter which is compiled during its build phase, so this is the best way to get an opendev python-base image with debug symbols. Change-Id: I1d89ac947cd3bea8a468f3ee022fb4cc93bece1f
2023-10-10 08:03:31 -07:00
- system-config-promote-image-python-base-3.11-bookworm-debug
Add python3.12 bookworm base images This is probably of limited utility currently as we don't have any good test platforms for running jobs on top of python3.12 outside of these container images. That said this will ensure our base job setup works with python3.12, and give early adopters the option of working with python3.12 in this way. Change-Id: Ief76c76e86c8cbae50e9fc7cb34356fa1c8e99a8
2023-10-18 13:38:14 -07:00
- system-config-promote-image-python-base-3.12-bookworm
- system-config-promote-image-python-base-3.12-bookworm-debug
Add Debian bookworm based python images uWSGI: libffi7 has been retired with bullseye so update to libffi8 Change-Id: I3e7e05b0e0148d73e7f0087f456a3f8de10e0c87
2023-07-11 12:32:18 +10:00
- system-config-promote-image-python-builder-3.10-bookworm
- system-config-promote-image-python-builder-3.11-bookworm
Add python3.12 bookworm base images This is probably of limited utility currently as we don't have any good test platforms for running jobs on top of python3.12 outside of these container images. That said this will ensure our base job setup works with python3.12, and give early adopters the option of working with python3.12 in this way. Change-Id: Ief76c76e86c8cbae50e9fc7cb34356fa1c8e99a8
2023-10-18 13:38:14 -07:00
- system-config-promote-image-python-builder-3.12-bookworm
Add Debian bookworm based python images uWSGI: libffi7 has been retired with bullseye so update to libffi8 Change-Id: I3e7e05b0e0148d73e7f0087f456a3f8de10e0c87
2023-07-11 12:32:18 +10:00
- system-config-promote-image-uwsgi-base-3.10-bookworm
- system-config-promote-image-uwsgi-base-3.11-bookworm
Add python3.12 bookworm base images This is probably of limited utility currently as we don't have any good test platforms for running jobs on top of python3.12 outside of these container images. That said this will ensure our base job setup works with python3.12, and give early adopters the option of working with python3.12 in this way. Change-Id: Ief76c76e86c8cbae50e9fc7cb34356fa1c8e99a8
2023-10-18 13:38:14 -07:00
- system-config-promote-image-uwsgi-base-3.12-bookworm
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
# NOTE: infra-prod-* jobs have a hierarchy below that ensure
# they can run in parallel. We are deliberately keeping their
# dependencies here rather than job definitions to help keep
# these relationships clear.
# This installs the ansible on bridge that all the infra-prod
# jobs will run with. Note the jobs use this ansible to then
# run against zuul's checkout of system-config.
Rename install-ansible to bootstrap-bridge This used to be called "bridge", but was then renamed with Ia7c8dd0e32b2c4aaa674061037be5ab66d9a3581 to install-ansible to be clearer. It is true that this is installing Ansible, but as part of our reworking for parallel jobs this is the also the synchronisation point where we should be deploying the system-config code to run for the buildset. Thus naming this "boostrap-bridge" should hopefully be clearer again about what's going on. I've added a note to the job calling out it's difference to the infra-prod-service-bridge job to hopefully also avoid some of the inital confusion. Change-Id: I4db1c883f237de5986edb4dc4c64860390cc8e22
2021-12-03 11:32:31 +11:00
- infra-prod-bootstrap-bridge
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
# From now on, all jobs should depend on base
- infra-prod-base: &infra-prod-base
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
dependencies:
Rename install-ansible to bootstrap-bridge This used to be called "bridge", but was then renamed with Ia7c8dd0e32b2c4aaa674061037be5ab66d9a3581 to install-ansible to be clearer. It is true that this is installing Ansible, but as part of our reworking for parallel jobs this is the also the synchronisation point where we should be deploying the system-config code to run for the buildset. Thus naming this "boostrap-bridge" should hopefully be clearer again about what's going on. I've added a note to the job calling out it's difference to the infra-prod-service-bridge job to hopefully also avoid some of the inital confusion. Change-Id: I4db1c883f237de5986edb4dc4c64860390cc8e22
2021-12-03 11:32:31 +11:00
- name: infra-prod-bootstrap-bridge
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
soft: true
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
# Legacy puppet hosts
- infra-prod-remote-puppet-else: &infra-prod-remote-puppet-else
dependencies:
- name: infra-prod-base
soft: true
#
# Only depends on base, or amongst themselves.
#
- infra-prod-service-bridge: &infra-prod-service-bridge
dependencies:
- name: infra-prod-base
soft: true
- infra-prod-run-cloud-launcher: &infra-prod-run-cloud-launcher
dependencies:
# depends on the cloud config written out by
# service-bridge
- name: infra-prod-service-bridge
soft: true
- infra-prod-service-kerberos: &infra-prod-service-kerberos
dependencies:
- name: infra-prod-base
soft: true
- infra-prod-service-afs: &infra-prod-service-afs
dependencies:
- name: infra-prod-base
soft: true
# NOTE(ianw) in theory we'd want auth changes before
# updating services like openafs using them. Not sure
# in practice this matters much; we very rarely change
# things here anyway.
- name: infra-prod-service-kerberos
soft: true
- infra-prod-service-nameserver: &infra-prod-service-nameserver
dependencies:
- name: infra-prod-base
soft: true
- infra-prod-service-mirror-update: &infra-prod-service-mirror-update
dependencies:
- name: infra-prod-base
soft: true
#
# Hosts using certificates and backups
#
# Hosts that backup should depend on this as this will create
# the users and deploy the keys required for the borg-backup
# role to work.
- infra-prod-service-borg-backup: &infra-prod-service-borg-backup
dependencies:
- name: infra-prod-base
soft: true
# Hosts that have letsencrypt certs should depend on this, as
# it will write out the key material before they try to start
# services that depend on it. For simplicity, we parent to
# this job.
- infra-prod-letsencrypt: &infra-prod-letsencrypt
dependencies:
- name: infra-prod-base
soft: true
- name: infra-prod-service-nameserver
soft: true
# letsencrypt depdencies. keep in alphabetical order
- infra-prod-service-codesearch: &infra-prod-service-codesearch
Migrate codesearch site to container The hound project has undergone a small re-birth and moved to https://github.com/hound-search/hound which has broken our deployment. We've talked about leaving codesearch up to gitea, but it's not quite there yet. There seems to be no point working on the puppet now. This builds a container than runs houndd. It's an opendev specific container; the config is pulled from project-config directly. There's some custom scripts that drive things. Some points for reviewers: - update-hound-config.sh uses "create-hound-config" (which is in jeepyb for historical reasons) to generate the config file. It grabs the latest projects.yaml from project-config and exits with a return code to indicate if things changed. - when the container starts, it runs update-hound-config.sh to populate the initial config. There is a testing environment flag and small config so it doesn't have to clone the entire opendev for functional testing. - it runs under supervisord so we can restart the daemon when projects are updated. Unlike earlier versions that didn't start listening till indexing was done, this version now puts up a "Hound is not ready yet" message when while it is working; so we can drop all the magic we were doing to probe if hound is listening via netstat and making Apache redirect to a status page. - resync-hound.sh is run from an external cron job daily, and does this update and restart check. Since it only reloads if changes are made, this should be relatively rare anyway. - There is a PR to monitor the config file (https://github.com/hound-search/hound/pull/357) which would mean the restart is unnecessary. This would be good in the near and we could remove the cron job. - playbooks/roles/codesearch is unexciting and deploys the container, certificates and an apache proxy back to localhost:6080 where hound is listening. I've combined removal of the old puppet bits here as the "-codesearch" namespace was already being used. Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
2020-11-17 17:13:46 +11:00
dependencies:
- name: infra-prod-letsencrypt
soft: true
- name: system-config-promote-image-hound
soft: true
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-eavesdrop: &infra-prod-service-eavesdrop
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
dependencies:
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- name: infra-prod-service-borg-backup
soft: true
- name: infra-prod-letsencrypt
soft: true
- name: system-config-promote-image-ircbot
soft: true
- name: system-config-promote-image-matrix-eavesdrop
soft: true
- infra-prod-service-etherpad: &infra-prod-service-etherpad
dependencies:
- name: infra-prod-service-borg-backup
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
soft: true
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- name: infra-prod-letsencrypt
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
soft: true
- name: system-config-promote-image-etherpad
soft: true
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-gitea: &infra-prod-service-gitea
dependencies:
- name: infra-prod-service-borg-backup
soft: true
- name: infra-prod-letsencrypt
soft: true
- name: system-config-promote-image-gitea
soft: true
- infra-prod-service-gitea-lb: &infra-prod-service-gitea-lb
dependencies:
Fix name for haproxy-statsd dependency Mixed up with gitea-lb naming. Fixes I19db98fcec5715c33b62c9c9ba5234fd55700fd8 Signed-off-by: Dr. Jens Harbott <harbott@osism.tech> Change-Id: I91d077102904a2144d12bc60eb7341f1065473b4
2021-12-01 17:30:39 +01:00
- name: system-config-promote-image-haproxy-statsd
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
soft: true
- infra-prod-service-grafana: &infra-prod-service-grafana
Grafana container deployment This uses the Grafana container created with Iddfafe852166fe95b3e433420e2e2a4a6380fc64 to run the grafana.opendev.org service. We retain the old model of an Apache reverse-proxy; it's well tested and understood, it's much easier than trying to map all the SSL termination/renewal/etc. into the Grafana container and we don't have to convince ourselves the container is safe to be directly web-facing. Otherwise this is a fairly straight forward deployment of the container. As before, it uses the graph configuration kept in project-config which is loaded in with grafyaml, which is included in the container. Once nice advantage is that it makes it quite easy to develop graphs locally, using the container which can talk to the public graphite instance. The documentation has been updated with a reference on how to do this. Change-Id: I0cc76d29b6911aecfebc71e5fdfe7cf4fcd071a4
2020-06-23 11:41:27 +10:00
dependencies:
- name: infra-prod-letsencrypt
soft: true
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-graphite: &infra-prod-service-graphite
dependencies:
- name: infra-prod-letsencrypt
soft: true
Add a keycloak server This adds a keycloak server so we can start experimenting with it. It's based on the docker-compose file Matthieu made for Zuul (see https://review.opendev.org/819745 ) We should be able to configure a realm and federate with openstackid and other providers as described in the opendev auth spec. However, I am unable to test federation with openstackid due its inability to configure an oauth app at "localhost". Therefore, we will need an actual deployed system to test it. This should allow us to do so. It will also allow use to connect realms to the newly available Zuul admin api on opendev. It should be possible to configure the realm the way we want, then export its configuration into a JSON file and then have our playbooks or the docker-compose file import it. That would allow us to drive change to the configuration of the system through code review. Because of the above limitation with openstackid, I think we should regard the current implementation as experimental. Once we have a realm configuration that we like (which we will create using the GUI), we can chose to either continue to maintain the config with the GUI and appropriate file backups, or switch to a gitops model based on an export. My understanding is that all the data (realms configuration and session) are kept in an H2 database. This is probably sufficient for now and even production use with Zuul, but we should probably switch to mariadb before any heavy (eg gerrit, etc) production use. This is a partial implementation of https://docs.opendev.org/opendev/infra-specs/latest/specs/central-auth.html We can re-deploy with a new domain when it exists. Change-Id: I2e069b1b220dbd3e0a5754ac094c2b296c141753 Co-Authored-By: Matthieu Huin <mhuin@redhat.com>
2021-11-30 13:03:12 -08:00
- infra-prod-service-keycloak: &infra-prod-service-keycloak
dependencies:
- name: infra-prod-letsencrypt
soft: true
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-meetpad: &infra-prod-service-meetpad
dependencies:
- name: infra-prod-letsencrypt
soft: true
Add a mailman3 list server This should now be a largely functional deployment of mailman 3. There are still some bits that need testing but we'll use followup changes to force failure and hold nodes. This deployment of mailman3 uses upstream docker container images. We currently hack up uids and gids to accomodate that. We also hack up the settings file and bind mount it over the upstream file in order to use host networking. We override the hyperkitty index type to xapian. All list domains are hosted in a single installation and we use native vhosting to handle that. We'll deploy this to a new server and migrate one mailing list domain at a time. This will allow us to start with lists.opendev.org and test things like dmarc settings before expanding to the remaining lists. A migration script is also included, which has seen extensive testing on held nodes for importing copies of the production data sets. Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
2022-07-27 11:35:41 -07:00
- infra-prod-service-lists3: &infra-prod-service-lists3
dependencies:
- name: infra-prod-service-borg-backup
soft: true
- name: infra-prod-letsencrypt
soft: true
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
- name: system-config-promote-image-mailman
soft: true
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-mirror: &infra-prod-service-mirror
dependencies:
- name: infra-prod-letsencrypt
soft: true
- infra-prod-service-nodepool: &infra-prod-service-nodepool
dependencies:
- name: infra-prod-letsencrypt
soft: true
- infra-prod-service-static: &infra-prod-service-static
dependencies:
- name: infra-prod-letsencrypt
soft: true
- infra-prod-service-paste: &infra-prod-service-paste
dependencies:
- name: infra-prod-service-borg-backup
soft: true
- name: infra-prod-letsencrypt
soft: true
- infra-prod-service-registry: &infra-prod-service-registry
dependencies:
- name: infra-prod-letsencrypt
soft: true
- infra-prod-service-refstack: &infra-prod-service-refstack
dependencies:
- name: infra-prod-service-borg-backup
soft: true
- name: infra-prod-letsencrypt
soft: true
- name: system-config-promote-image-refstack
soft: true
- infra-prod-service-review: &infra-prod-service-review
dependencies:
- name: infra-prod-service-borg-backup
soft: true
- name: infra-prod-letsencrypt
soft: true
Update gerrit image build job dependencies We just upgraded Gerrit to 3.10 and need to update the job dependencies for infra-prod-service-review and infra-prod-manage-projects to match. This probably should've been done in the prior change that updated the gerrit docker image to 3.10. Maybe we'll remember to do these updates together when we upgrade to 3.11. Change-Id: Ie72652f7ddf88f82608d74b033d22bebbf5339e2
2024-12-06 11:33:18 -08:00
- name: system-config-promote-image-gerrit-3.10
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
soft: true
Add Jaeger tracing server Change-Id: I1aa68b1d5f99364fa09776301894b922ed169a3a
2022-09-05 13:42:18 -07:00
- infra-prod-service-tracing: &infra-prod-service-tracing
dependencies:
- name: infra-prod-letsencrypt
soft: true
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-zookeeper: &infra-prod-service-zookeeper
dependencies:
- name: infra-prod-letsencrypt
soft: true
- name: system-config-promote-image-zookeeper-statsd
infra-prod: fix infra-prod-service-zookeeper soft dependency This is a typo from the job shuffle in I8f6150ec2f696933c93560c11fed0fd16b11bf65 -- this should be a soft dependency. It is currently causing periodic jobs to fail Change-Id: Ia420e74a1d64b12b63b1697e61992c46119451dc
2021-12-13 11:01:45 +11:00
soft: true
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-zuul: &infra-prod-service-zuul
dependencies:
- name: infra-prod-service-borg-backup
soft: true
- name: infra-prod-letsencrypt
soft: true
# should reconfigure after any project updates
- name: infra-prod-manage-projects
soft: true
Add a standalone zuul db server Change-Id: Ibb260f820dbc1d9d6ca523ff3903134612cb003e
2024-04-04 09:07:19 -07:00
- infra-prod-service-zuul-db
Add Zuul load balancer This adds a load balancer for zuul-web and fingergw. Change-Id: Id5aa01151f64f3c85e1532ad66999ef9471c5896
2022-02-10 09:28:46 -08:00
- infra-prod-service-zuul-lb: &infra-prod-service-zuul-lb
dependencies:
- name: system-config-promote-image-haproxy-statsd
soft: true
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-zuul-preview: &infra-prod-service-zuul-preview
dependencies:
- name: infra-prod-letsencrypt
soft: true
#
# Jobs that run as secondary steps
#
# accessbot should run on a setup eavesdrop host
- infra-prod-run-accessbot: &infra-prod-run-accessbot
dependencies:
- name: infra-prod-base
soft: true
- name: infra-prod-service-eavesdrop
soft: true
Run accessbot less frequently We already run accessbot in project-config when the accessbot script changes. We don't need to run it whenever any of the puppet or other config on eavesdrop runs, not do we need to run it hourly. Just run it nightly and on changes to the actual accessbot config. Change-Id: Idd47f7c96f677fd1e1c8da3be262a52a70646acd
2020-05-08 08:12:37 -05:00
- name: system-config-promote-image-accessbot
soft: true
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
# manage-projects runs jeepyb etc. and should run on
# a setup review host. also sets up gitea
- infra-prod-manage-projects: &infra-prod-manage-projects
dependencies:
- name: infra-prod-base
soft: true
- name: infra-prod-service-review
soft: true
- name: infra-prod-service-gitea
soft: true
Update gerrit image build job dependencies We just upgraded Gerrit to 3.10 and need to update the job dependencies for infra-prod-service-review and infra-prod-manage-projects to match. This probably should've been done in the prior change that updated the gerrit docker image to 3.10. Maybe we'll remember to do these updates together when we upgrade to 3.11. Change-Id: Ie72652f7ddf88f82608d74b033d22bebbf5339e2
2024-12-06 11:33:18 -08:00
- name: system-config-promote-image-gerrit-3.10
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
soft: true
# Note that this job also runs from project-config, so we
# match system-config specific files here rather than the
# job definition.
files:
- inventory/.*
- playbooks/manage-projects.yaml
- inventory/service/group_vars/review.yaml
- inventory/service/group_vars/gitea.yaml
- inventory/service/host_vars/gitea
- inventory/service/host_vars/review
- playbooks/roles/gitea-git-repos/
- playbooks/roles/gerrit/defaults/main.yaml
- playbooks/roles/gerrit/tasks/manage-projects.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
periodic:
jobs:
- developer-openstack-goaccess-report
- docs-opendev-goaccess-report
- docs-openstack-goaccess-report
- docs-starlingx-goaccess-report
- governance-openstack-goaccess-report
- releases-openstack-goaccess-report
- security-openstack-goaccess-report
- specs-openstack-goaccess-report
- tarballs-opendev-goaccess-report
- zuul-ci-goaccess-report
# Nightly runs of ansible things for catchup
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
# Keep in order from above
Rename install-ansible to bootstrap-bridge This used to be called "bridge", but was then renamed with Ia7c8dd0e32b2c4aaa674061037be5ab66d9a3581 to install-ansible to be clearer. It is true that this is installing Ansible, but as part of our reworking for parallel jobs this is the also the synchronisation point where we should be deploying the system-config code to run for the buildset. Thus naming this "boostrap-bridge" should hopefully be clearer again about what's going on. I've added a note to the job calling out it's difference to the infra-prod-service-bridge job to hopefully also avoid some of the inital confusion. Change-Id: I4db1c883f237de5986edb4dc4c64860390cc8e22
2021-12-03 11:32:31 +11:00
- infra-prod-bootstrap-bridge
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-base: *infra-prod-base
- infra-prod-remote-puppet-else: *infra-prod-remote-puppet-else
- infra-prod-letsencrypt: *infra-prod-letsencrypt
- infra-prod-service-bridge: *infra-prod-service-bridge
- infra-prod-run-cloud-launcher: *infra-prod-run-cloud-launcher
- infra-prod-service-kerberos: *infra-prod-service-kerberos
- infra-prod-service-afs: *infra-prod-service-afs
- infra-prod-service-nameserver: *infra-prod-service-nameserver
- infra-prod-service-mirror-update: *infra-prod-service-mirror-update
- infra-prod-service-borg-backup: *infra-prod-service-borg-backup
- infra-prod-letsencrypt: *infra-prod-letsencrypt
- infra-prod-service-codesearch: *infra-prod-service-codesearch
- infra-prod-service-eavesdrop: *infra-prod-service-eavesdrop
- infra-prod-service-etherpad: *infra-prod-service-etherpad
- infra-prod-service-gitea: *infra-prod-service-gitea
- infra-prod-service-gitea-lb: *infra-prod-service-gitea-lb
- infra-prod-service-grafana: *infra-prod-service-grafana
- infra-prod-service-graphite: *infra-prod-service-graphite
Add a keycloak server This adds a keycloak server so we can start experimenting with it. It's based on the docker-compose file Matthieu made for Zuul (see https://review.opendev.org/819745 ) We should be able to configure a realm and federate with openstackid and other providers as described in the opendev auth spec. However, I am unable to test federation with openstackid due its inability to configure an oauth app at "localhost". Therefore, we will need an actual deployed system to test it. This should allow us to do so. It will also allow use to connect realms to the newly available Zuul admin api on opendev. It should be possible to configure the realm the way we want, then export its configuration into a JSON file and then have our playbooks or the docker-compose file import it. That would allow us to drive change to the configuration of the system through code review. Because of the above limitation with openstackid, I think we should regard the current implementation as experimental. Once we have a realm configuration that we like (which we will create using the GUI), we can chose to either continue to maintain the config with the GUI and appropriate file backups, or switch to a gitops model based on an export. My understanding is that all the data (realms configuration and session) are kept in an H2 database. This is probably sufficient for now and even production use with Zuul, but we should probably switch to mariadb before any heavy (eg gerrit, etc) production use. This is a partial implementation of https://docs.opendev.org/opendev/infra-specs/latest/specs/central-auth.html We can re-deploy with a new domain when it exists. Change-Id: I2e069b1b220dbd3e0a5754ac094c2b296c141753 Co-Authored-By: Matthieu Huin <mhuin@redhat.com>
2021-11-30 13:03:12 -08:00
- infra-prod-service-keycloak: *infra-prod-service-keycloak
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-meetpad: *infra-prod-service-meetpad
Add a mailman3 list server This should now be a largely functional deployment of mailman 3. There are still some bits that need testing but we'll use followup changes to force failure and hold nodes. This deployment of mailman3 uses upstream docker container images. We currently hack up uids and gids to accomodate that. We also hack up the settings file and bind mount it over the upstream file in order to use host networking. We override the hyperkitty index type to xapian. All list domains are hosted in a single installation and we use native vhosting to handle that. We'll deploy this to a new server and migrate one mailing list domain at a time. This will allow us to start with lists.opendev.org and test things like dmarc settings before expanding to the remaining lists. A migration script is also included, which has seen extensive testing on held nodes for importing copies of the production data sets. Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
2022-07-27 11:35:41 -07:00
- infra-prod-service-lists3: *infra-prod-service-lists3
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-mirror: *infra-prod-service-mirror
- infra-prod-service-nodepool: *infra-prod-service-nodepool
- infra-prod-service-static: *infra-prod-service-static
- infra-prod-service-paste: *infra-prod-service-paste
- infra-prod-service-registry: *infra-prod-service-registry
- infra-prod-service-refstack: *infra-prod-service-refstack
- infra-prod-service-review: *infra-prod-service-review
Add Jaeger tracing server Change-Id: I1aa68b1d5f99364fa09776301894b922ed169a3a
2022-09-05 13:42:18 -07:00
- infra-prod-service-tracing: *infra-prod-service-tracing
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-zookeeper: *infra-prod-service-zookeeper
- infra-prod-service-zuul: *infra-prod-service-zuul
Add a standalone zuul db server Change-Id: Ibb260f820dbc1d9d6ca523ff3903134612cb003e
2024-04-04 09:07:19 -07:00
- infra-prod-service-zuul-db
Add Zuul load balancer This adds a load balancer for zuul-web and fingergw. Change-Id: Id5aa01151f64f3c85e1532ad66999ef9471c5896
2022-02-10 09:28:46 -08:00
- infra-prod-service-zuul-lb: *infra-prod-service-zuul-lb
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-zuul-preview: *infra-prod-service-zuul-preview
- infra-prod-run-accessbot: *infra-prod-run-accessbot
- infra-prod-manage-projects: *infra-prod-manage-projects
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
opendev-prod-hourly:
jobs:
Rename install-ansible to bootstrap-bridge This used to be called "bridge", but was then renamed with Ia7c8dd0e32b2c4aaa674061037be5ab66d9a3581 to install-ansible to be clearer. It is true that this is installing Ansible, but as part of our reworking for parallel jobs this is the also the synchronisation point where we should be deploying the system-config code to run for the buildset. Thus naming this "boostrap-bridge" should hopefully be clearer again about what's going on. I've added a note to the job calling out it's difference to the infra-prod-service-bridge job to hopefully also avoid some of the inital confusion. Change-Id: I4db1c883f237de5986edb4dc4c64860390cc8e22
2021-12-03 11:32:31 +11:00
- infra-prod-bootstrap-bridge
Refactor infra-prod jobs for parallel running Refactor the infra-prod jobs to specify dependencies so they can run in parallel. Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-10-21 14:43:48 +11:00
- infra-prod-service-bridge: *infra-prod-service-bridge
- infra-prod-service-nodepool: *infra-prod-service-nodepool
- infra-prod-service-registry: *infra-prod-service-registry
- infra-prod-service-zuul: *infra-prod-service-zuul
- infra-prod-service-eavesdrop: *infra-prod-service-eavesdrop
Copy Permalink