This does local backups of the nodepool zk image image data to
/var/log/nodepool on the nodepool-builders. These hosts don't get
offsite backups but we run mutliple redundant servers. This data isn't
critical as we can start from scratch, but may be useful if we don't
want to go through all that trouble.
Change-Id: I7d150df9c0d9566ef2d32167cea535e29822cfa2
podman, used by the new containerfile element, requires a
non-overlayfs volume at /var/lib/containers to be able to start and
extract the container images for us to build from. Add a separate
volume for this.
Change-Id: I6629034ad0b300d392d3d989dbbf17a1343c06e1
Add a variable to configure upload-workers for nodepool-builder
daemons.
Reduce our defaults for nb03 to see if we can get more reliable
uploads.
Change-Id: I819bdd262c7118cbde4e6ffdc12aa3ac64569a96
So we can stop/pull/start, move the pull tasks to their own files
and add a playbook that invokes them.
Change-Id: I4f351c1d28e5e4606e0a778e545a3a805525ac71
By default docker-compose sends a sigterm to containers when they are
stopping. It then waitsfor 10 seconds before sending a sigkill. Our
nodepool-builder containers are restarted when new images are available
and if that happens during a dib build we leak contents into dib_tmp.
One theory is that we aren't giving dib enough time to clean up after
itself so increase the 10 second period before sending sigkill to 90
seconds.
I'm not sure if this will actually help, but it can't hurt much. If the
processes die quicker we don't go any slower and if they don't die
quicker then we're giving them more time to clean up.
Change-Id: Id12cac89cccfc14a8d262e8f8494046df777a80a
We are running with the default upload workers count of 4 which is half
of our previous ansible/puppet valud of 8 (we have 8 vcpus on these
servers). Increase the worker count to 8 to improve upload rate.
Change-Id: I3c051968acc8c32711cd7063469d4a80077ba587
This avoids the conflict with the zuul user (1000) on the test
nodes. The executor will continue to use the default username
of 'zuul' as the ansible_user in the inventory.
This change also touches the zk and nodepool deployment to use
variables for the usernames and uids to make changes like this
easier. No changes are intended there.
Change-Id: Ib8cef6b7889b23ddc65a07bcba29c21a36e3dcb5
We noticed that our zuul scheduler was running out of disk and one of
the causes of this is we are pulling all of the wonderful new zuul
images and not pruning them. This happens because we were only pruning
when (re)starting services and we don't do that automatically with Zuul.
Address this by always pruning after pulling even if we don't restart
services. This should be safe because prune will leave the latest tagged
images as well as the running images.
This should keep our disk consumption down.
Change-Id: Ibdd22ac42d86781f1e87c3d11e05fd8f99677167
This should be set for each host's name; this looks like it was just
not templated correctly from the initial commit
I230f5291e0bd928af2e00966d76c3f385b749cb6.
Change-Id: If86ee21268c0fe6bb60c61750f551db89234ed0e
We're running these in containers now. Please not to try to start
them the old way.
failed_when false is because we can't disable the old service
in the gate if there is no service file installed.
Change-Id: Ia4560f385fc98e23f987a67a1dfa60c3188816b6
We don't want to run it as a daemon now that we're passing
logging config to nodepool. Pass -f - since that's what
the Dockerfile does.
Change-Id: I87b4210ee7e5a622a34944c6345008082e75145b
We have a logging config to log to /var/log/nodepool but we weren't
using it. Start using it.
Add logging config to nodepool-builder
We should log nodepool builder to /var/log/nodepool too.
Change-Id: I6e7196dc12e8c1bfc54274432b94cf53629bdf3d
We want to use stop_grace_period to manage gerrit service stops. This
feature was added in docker-compose 1.10 but the distro provides 1.5.
Work around this by installing docker-compose from pypi.
This seems like a useful feature and we want to manage docker-compose
the same way globally so move docker-compose installation into the
install-docker role.
New docker-compose has slightly different output that we must check for
in the gitea start/stop machinery. We also need to check for different
container name formatting in our test cases. We should pause here and
consider if this has any upgrade implications for our existing services.
Change-Id: Ia8249a2b84a2ef167ee4ffd66d7a7e7cff8e21fb
We don't actually need version 3. Mark it as version 2 to keep it
inline with everything else. In general we should only increase
past v2 if we need a specific feature.
Change-Id: Ie243da369ddec30e0eca4805434d572e12c40491
Currently we deploy the openstacksdk config into ~nodepool/.config on
the container, and then map this directory back to /etc/openstack in
the docker-compose. The config-file still hard-codes the
limestone.pem file to ~nodepool/.config.
Switch the nodepool-builder_opendev group to install to
/etc/openstack, and update the nodepool config file template to use
the configured directory for the .pem path.
Also update the testing paths.
Story: #2007407
Task: #39015
Change-Id: I9ca77927046e2b2e3cee9a642d0bc566e3871515
We rolled out review-dev with podman and it worked fine for us. It
worked less fine for nodepool-builder, although we still might be
able to solve it. Maybe right now isn't the time to do this switch.
Gitea, gitea-lb and zuul-registry all use docker instead of podman.
The only thing running with podman right now is review-dev. We can
do a manual cleanup of podman there before runnign this to keep
things simple:
- stop gerrit service
- uninstall podman and podman-compose
- uninstall podman ppa config
- uninstall pip3
Then let ansible install docker and docker compose up.
Story: #2007407
Task: #39062
Change-Id: I9bf99b18559d49d11ba99a96f02a4a45a4f65a86
This was missing but is part of the required runtime directories for
the container (for now, until we maybe move all this to volumes).
Change-Id: I9e173eb799026520588722caaf60a160abc6b130
