Run service-bridge in zuul and semaphore everything

Make a base job for the various service playbooks to capture the
fact that we should run these after update-system-config, and if
they run, after both install-ansible and base.

Attach a semphore to the base job, because while many of the
playbooks should be independent, some may not be, and we need to
make sure things don't double-run in periodic and promote.

Transition service-bridge from run_all to zuul, basing it on the
new base job.

While we're in here, reduce manage-projects forks to 10, because
let's face it, that's a more sensible number when there aren't
that many hosts.

Change-Id: I22e9edaea75dcfdab56f667f7c93cdd3ee25406c

.zuul.yaml

@@ -1395,6 +1395,11 @@
           '/etc/apache2/': logs
           '/var/log/apache2/': logs
 
+# Make sure only one run of a system-config playbook happens at a time
+- semaphore:
+    name: infra-prod-playbook
+    max: 1
+
 - job:
     name: infra-prod-playbook
     description: |
@@ -1405,6 +1410,7 @@
       specify the playbook relative to
       bridge.openstack.org:/opt/system-config/playbooks
     abstract: true
+    semaphore: infra-prod-playbook
     run: playbooks/zuul/run-production-playbook.yaml
     vars:
       ansible_forks: 5
@@ -1423,10 +1429,6 @@
     vars:
       playbook_name: zuul_reconfigure.yaml
 
-# Make sure only one run of manage-projects happens at a time
-- semaphore:
-    name: infra-prod-manage-projects-runs
-    max: 1
 
 - job:
     name: infra-prod-update-system-config
@@ -1485,10 +1487,40 @@
     allowed-projects:
       - opendev/system-config
       - openstack/project-config
-    semaphore: infra-prod-manage-projects-runs
     vars:
       playbook_name: manage-projects.yaml
-      ansible_forks: 50
+      ansible_forks: 10
+
+- job:
+    name: infra-prod-service-base
+    parent: infra-prod-playbook
+    description: Base job for most service playbooks
+    abstract: true
+    dependencies:
+      - name: infra-prod-update-system-config
+      - name: infra-prod-install-ansible
+        soft: true
+      - name: infra-prod-base
+        soft: true
+
+- job:
+    name: infra-prod-service-bridge
+    parent: infra-prod-service-base
+    description: Run service-bridge.yaml playbook
+    semaphore: infra-prod-service-bridge
+    vars:
+      playbook_name: service-bridge.yaml
+    files:
+      - playbooks/service-bridge.yaml
+      - playbooks/host_vars/bridge.openstack.org.yaml
+      - playbooks/roles/ansible-cron/.*
+      - playbooks/roles/cloud-launcher-cron/.*
+      - playbooks/roles/edit-secrets-script/.*
+      - playbooks/roles/install-kubectl/.*
+      - playbooks/roles/configure-kubectl/.*
+      - playbooks/roles/configure-openstacksdk/.*
+      - playbooks/templates/clouds/bridge_clouds.yaml.j2
+      - playbooks/templates/clouds/bridge_all_clouds.yaml.j2
 
 - job:
     name: system-config-goaccess-report
@@ -1837,6 +1869,7 @@
               - name: infra-prod-update-system-config
               - name: infra-prod-install-ansible
                 soft: true
+        - infra-prod-service-bridge
     periodic:
       jobs:
         - developer-openstack-goaccess-report
@@ -1853,3 +1886,4 @@
         - infra-prod-update-system-config
         - infra-prod-install-ansible
         - infra-prod-base
+        - infra-prod-service-bridge

run_all.sh

@@ -72,11 +72,6 @@ set +e
 # Run all the ansible playbooks under timeout to prevent them from getting
 # stuck if they are oomkilled
 
-# Service playbooks
-start_timer
-timeout -k 2m 30m ansible-playbook -f 50 ${ANSIBLE_PLAYBOOKS}/service-bridge.yaml
-send_timer service-bridge
-
 start_timer
 timeout -k 2m 30m ansible-playbook -f 50 ${ANSIBLE_PLAYBOOKS}/service-gitea-lb.yaml
 send_timer gitea-lb