291 Commits

Author SHA1 Message Date
Zuul
99f809ccc5 Merge "Use zuul checkouts of ansible roles from other repos" 2020-05-07 18:41:21 +00:00
Zuul
b59c9d8a2b Merge "Retire nb01/02.openstack.org" 2020-05-07 18:00:10 +00:00
Zuul
502ddff9b3 Merge "Test zuul-executor on focal" 2020-05-07 17:53:20 +00:00
Ian Wienand
e400865dd0 Retire nb01/02.openstack.org
Remove references to these older builders.  We thank them for their
service.

Change-Id: I1f48f070406bee79ac0d1de61beb44eb7d58d605
2020-05-07 13:06:26 +10:00
Ian Wienand
f204337268 Add nb01/nb02 opendev servers
These are replacements for the nb01/02.openstack.org puppet servers

Change-Id: I376d70ee375289b004fb859751743c6fafa21411
2020-05-07 09:10:26 +10:00
Monty Taylor
39495ffdd5 Test zuul-executor on focal
We want to replace the current executors with focal executors.
Make sure zuul-executor can run there.

Kubic is apparently the new source for libcontainers stuff:

  https://podman.io/getting-started/installation.html

Use only timesyncd on focal

ntp and timesyncd have a hard conflict with each other. Our test
images install ntp. Remove it and just stay with timesyncd.

Change-Id: I0126f7c77d92deb91711f38a19384a9319955cf5
2020-05-06 18:00:29 -05:00
Monty Taylor
4b9d1a88bd Use zuul checkouts of ansible roles from other repos
We have two standalone roles, puppet and cloud-launcher, but we
currently install them with galaxy so depends-on patches don't
work. We also install them every time we run anything, even if
we don't need them for the playbook in question.

Add two roles, one to install a set of ansible roles needed by
the host in question, and the other to encapsulate the sequence
of running puppet, which now includes installing the puppet
role, installing puppet, disabling the puppet agent and then
running puppet.

As a followup, we'll do the same thing with the puppet modules,
so that we arent' cloning and rsyncing ALL of the puppet modules
all the time no matter what.

Change-Id: I69a2e99e869ee39a3da573af421b18ad93056d5b
2020-04-30 12:39:12 -05:00
Monty Taylor
e0619f17f1 Run nodepool launchers with ansible and containers
We don't run start in prod normally but we do need to run
it in the gate.

Change-Id: Iec50684280409eb978bf5638bf74ae16fad8aa26
2020-04-30 17:37:22 +00:00
Monty Taylor
8d7075b02f Run zookeeper cluster in nodepool jobs
Rather than running a local zookeeper, just run a real zookeeper.
Also, get rid of nb01-test and just use nb04 - what could possibly
go wrong?

Dynamically write zookeeper host information to nodepool.yaml

So that we can run an actual zk using the new zk role on hosts in
ansible inventory, we need to write out the ip addresses of the
hosts that we build in zuul. This means having the info baked in
to the file in project-config isn't going to work.

We can do this in prod too, it shouldn't hurt anything.

Increase timeout for run-service-nodepool

We need to fix the playbook, but we'll do that after we get the
puppet gone.

Change-Id: Ib01d461ae2c5cec3c31ec5105a41b1a99ff9d84a
2020-04-29 16:18:25 -05:00
Zuul
b21a8e58cf Merge "Run Zuul using Ansible and Containers" 2020-04-24 16:31:42 +00:00
Monty Taylor
f0b77485ec Run Zuul using Ansible and Containers
Zuul is publishing lovely container images, so we should
go ahead and start using them.

We can't use containers for zuul-executor because of the
docker->bubblewrap->AFS issue, so install from pip there.

Don't start any of the containers by default, which should
let us safely roll this out and then do a rolling restart.
For things (like web or mergers) where it's safe to do so,
a followup change will swap the flag.

Change-Id: I37dcce3a67477ad3b2c36f2fd3657af18bc25c40
2020-04-24 09:18:44 -05:00
Monty Taylor
d5c68c5131 Split codesearch into its own playbook
Make a service playbook, manifest and jobs for codesearch.

Remove openstack_project::server - it doesn't do anything.

Change-Id: I44c140de4ae0b283940f8e23e8c47af983934471
2020-04-21 13:18:28 -05:00
Monty Taylor
711295e918 Remove old etherpad.openstack.org
Once the DNS is swapped over to point at etherpad.opendev.org
we can delete the old stuff.

Change-Id: I626dd22b22a23619fcf460533336f1ddfec615d9
2020-04-19 10:58:46 -05:00
James E. Blair
42574b2b37 Run ZK from containers
Migration plan:
* add zk* to emergency
* copy data files on each node to a safe place for DR backup
* make a json data backup: zk-shell localhost:2181 --run-once 'mirror / json://!tmp!zookeeper-backup.json/'
* manually run a modified playbook to set up the docker infra without starting containers
* rolling restart; for each node:
  * stop zk
  * split data and log files and move them to new locations
  * remove zk packages
  * start zk containers
* remove from emergency; land this change.

Change-Id: Ic06c9cf9604402aa8eb4bb79238021c14c5d9563
2020-04-17 08:43:09 -07:00
Zuul
135a6a721e Merge "Back up a single gitea backend" 2020-04-14 20:33:27 +00:00
Monty Taylor
2ee77458a8 Back up a single gitea backend
We need to keep at least one of these databases.

Change-Id: Ic734498fbada70856f62de972d7863df472966e5
2020-04-13 08:53:16 -05:00
Monty Taylor
428c423548 Turn backup server back off
Change-Id: I988d6391672053e87722b2f0a10e98c0fa783c40
2020-04-10 13:46:29 -05:00
Monty Taylor
59679d009b Run ansible on the backup server
We need to pulse on the backup server to register etherpad.opendev.org.

Change-Id: Iaec41b1183373bd832dae70af4ae04dfb5bde263
2020-04-10 13:46:29 -05:00
Monty Taylor
ca5549fc6c Add review and etherpad to backup group
We should probably back these up.

Change-Id: I1e174273faefacea98ebece7a90a1baf93d52245
2020-04-10 13:46:25 -05:00
Monty Taylor
b23515c623 Make a new dockerized etherpad.opendev.org
Upstream likes building the settings file into the image, but that's
less exciting, let's bind-mount ours in.

Depends-On: https://review.opendev.org/717491/
Change-Id: Ia1894d884ef2a84e1282345b77fe07bf8898f367
2020-04-07 11:10:57 -05:00
Zuul
1fd2e226ab Merge "Remove inventory references to <static|files>.openstack.org" 2020-03-31 21:47:47 +00:00
Ian Wienand
476c3ac6f2 Remove inventory references to <static|files>.openstack.org
These hosts have been removed; remove the old references and
unnecessary groups, add the new host to cacti.

Change-Id: Ibcfd78a37e20e514c190ef801c2d44320c8b3f74
Story: #2006598
2020-04-01 07:49:02 +11:00
Zuul
70e2828ce4 Merge "Remove files from letsencrypt group" 2020-03-31 07:39:36 +00:00
James E. Blair
ecd3ecb4d4 Add meetpad01 to inventory
Change-Id: I3accead9f3a6f429574e03492cfbd4883a42a4bc
2020-03-27 10:02:31 -07:00
Zuul
ce3a064133 Merge "Add meetpad server" 2020-03-27 14:44:30 +00:00
Monty Taylor
a72ad58d5a Remove files from letsencrypt group
It got missed in an earlier cleanup.

Change-Id: If795fcb6637492518fe2ca2cd37ca6cb41afb101
2020-03-26 07:19:37 -05:00
Ian Wienand
dcc23be55b Remove static site puppet
Remove the puppet related to the static server

Story: #2006598
Task: #36758

Change-Id: I744da83d811e715ff8049eacf3013cd4831e1c49
2020-03-26 10:36:16 +11:00
Ian Wienand
f55580fbf0 Remove files02.openstack.org and related puppet
All this has moved to static.opendev.org; the server can now be
removed.

Change-Id: I8ca5d7a206e950c28bb8372a85b6a62d6b9ba00c
2020-03-26 10:36:13 +11:00
James E. Blair
8b093dacd5 Add meetpad server
Depends-On: https://review.opendev.org/714189
Change-Id: I5863aaa805a18f9085ee01c3205b0f9ad602922d
2020-03-25 07:44:24 -07:00
Monty Taylor
d3c8c1077b Switch to running gerrit via ansible+containers
This should be mostly a no-op - but we will need to do a shutdown
in emergency mode.

Tell the gerrit role to not run compose up when run as part of
remote_puppet_git.

Change-Id: Id45376c2697656a12afeacf317b6f26c85c08dad
2020-03-19 17:21:39 -05:00
Ian Wienand
ba5d9547c8 Replace nb01.opendev.org with nb04
We are starting over with the container nodepool host, and also moving
it to "nb04" to avoid any possibility of conflicting with the
short-hostname of nb01.openstack.org.

Story: #2007407
Task: #39064

Depends-On: https://review.opendev.org/713575
Depends-On: https://review.opendev.org/713571

Change-Id: I18ab9834ad4da201774e0abef56f618cd7839d36
2020-03-19 07:41:56 +11:00
Ian Wienand
dbe0bf1ee6 Add nb01.opendev.org
This configures an opendev nodepool-builder

Change-Id: Id8603d9d7caaac0a1ab935e1c7c80d32b02ae23e
Depends-On: https://review.opendev.org/693118
2020-03-11 09:16:31 +11:00
Ian Wienand
b1bfee423b nodepool-builder: Add webserver
This adds the webserver that serves the logs and generated images.

Change-Id: I230f5291e0bd928af2e00966d76c3f385b749cb6
2020-03-11 09:16:31 +11:00
Zuul
81c158c52f Merge "Add initial Ansible for nodepool hosts" 2020-03-10 00:10:36 +00:00
Jeremy Stanley
4cbdc2fc4d Set up inventory and cert for Open Edge mirror
This adds the Open Edge (formerly Fortnebula) CI mirror.

Change-Id: I1ccf2a602f8a41e00bc64a9516a326cc07d9b254
Depends-On: https://review.opendev.org/711787
2020-03-07 00:24:20 +00:00
Jeremy Stanley
43ed9fc297 Moving FortNebula to OpenEdge
Sister change for Ia5caff34d3fafaffc459e7572a4eef6bd94422ea and
removing earlier references to the mirror server in preparation for
building and adding the new one.

Change-Id: I7d506be85326835d5e77a0c9c461f2d457b1dfd3
2020-03-06 20:43:56 +00:00
Ian Wienand
281425a44d Add initial Ansible for nodepool hosts
This is a start at ansible-deployed nodepool environments.

We rename the minimal-nodepool element to nodepool-base-legacy, and
keep running that for the old nodes.

The groups are updated so that only the .openstack.org hosts will run
puppet.  Essentially they should remain unchanged.

We start a nodepool-base element that will replace the current
puppet-<openstackci|nodepool> deployment parts.  For step one, this
grabs project-config and links in the elements and config file.

A testing host is added for gate testing which should trigger these
roles.  This will build into a full deployment test of the builder
container.

Change-Id: If0eb9f02763535bf200062c51a8a0f8793b1e1aa
Depends-On: https://review.opendev.org/#/c/710700/
2020-03-06 14:02:52 +11:00
Zuul
75693aa43b Merge "Remove review-dev01.openstack.org" 2020-02-12 18:12:51 +00:00
Zuul
3308e83885 Merge "Get LE certs for review.o.o" 2020-02-12 18:12:50 +00:00
Monty Taylor
23b0667c45 Remove review-dev01.openstack.org
We've got a shiny new opendev one of these now, get rid of the
openstack.org one.

Change-Id: I9f5141a6ff23ab035ecedf100a417989453bfc1e
2020-02-12 14:40:46 +00:00
Monty Taylor
083cbf2911 Get LE certs for review.o.o
We have LE dns entries for review.o.o, but we're not actually
requesting the cert. Go ahead and request it - it'll make the
apache config easier to sort out.

Get the openstack.org certs for review-dev while we're at it.

Change-Id: I91d06c97993ba37204bd1fc326ae823e1b9c0c1a
Depends-On: https://review.opendev.org/707267
Depends-On: https://review.opendev.org/707255
2020-02-11 17:01:43 -06:00
Clark Boylan
0463609584 Add airship-citycloud mirror
This adds a mirror to the new airship citycloud region. Add the host to
the inventory and add necessary host vars for LE setup.

Depends-On: https://review.opendev.org/706573
Change-Id: I33cefe914911b4f5ce5e09e0329ba48e039ede64
2020-02-07 08:55:40 -08:00
Ian Wienand
08707baff2 Replace nb03.openstack.org
Due to persistent, unresolved network issues between the London and US
cloud (that don't appear to happen the other way), we have decided on
a hard Brexit for nb03.o.o and started a new server in the US cloud :)

Change-Id: I6557a9f272351578216bc525b6ddaffcf625f9f3
2020-02-07 15:09:20 +11:00
Monty Taylor
cc619fe589 Add review-dev01.opendev.org
Add a new review-dev server on the opendev domain with LE support
enabled.

Depends-On: https://review.opendev.org/705661
Change-Id: Ie32124cd617e9986602301f230e83bb138524fdf
2020-02-05 09:58:25 -06:00
Ian Wienand
7ce0d0fb32 Add static01.opendev.org
Add this host for serving content from AFS.

The

 _acme-challenge.governance.openstack.org
 _acme-challenge.security.openstack.org

CNAMES should be in place for creating the certificates (added with
Ie1b92f06b71aa6069fe831b26ba1cc272ce4562c).

Also add a cert for the base server (static.opendev.org) since we
added the DNS entries for it.

Change-Id: I55e0ac7487b02f9a816ac486ed01b73f82b391a5
Story: #2006598
Task: #37757
Depends-On: https://review.opendev.org/704469
2020-01-28 11:30:18 +11:00
Clark Boylan
cb00b967f7 Add new xenial status.openstack.org
Change-Id: I1876517992e5ab16b58fe9152458deb49c2ad807
2020-01-27 13:09:02 -08:00
Ian Wienand
9a75422145 Add linaro-us mirror
Change-Id: I4abf249fd385872f8ba56c3e41f99d160a68efcd
Story: #2007195
Task: #38358
Depends-On: https://review.opendev.org/703746
2020-01-22 20:17:53 +11:00
Ian Wienand
c3c96d3797 Add Linaro US cloud
Add the credentials for the newly provisioned us.linaro.cloud cloud

Change-Id: I0b81a8eeabec4e0b00258dc4e499c1d449b21681
2020-01-22 06:44:01 +11:00
Ian Wienand
f5b5ee9336 Add roles for a basic static server
Basic implementation of the opendev static server, described in

 https://docs.opendev.org/opendev/infra-specs/latest/specs/retire-static.html

Change-Id: Ie1b92f06b71aa6069fe831b26ba1cc272ce4562c
Story: #2006598
Task:  #37757
2020-01-16 14:10:08 +11:00
Monty Taylor
6f3a2792cc Switch to ansible on review-dev
The review-dev service playbook should do everything now that
the puppet did. Update how we're running things.

Change-Id: I70303c48328ea6713c24bf9c6f63d4808d30b95c
2020-01-14 12:04:15 -06:00