This fixes the zuul debug log's logrotate filename. We also increase the
rotation count to 30 daily logs for all zuul scheduler zuul processes
(this matches the old server).
We also create a /var/lib/zuul/backup dir so that status.json backups
have a location they can write to. We do this in the base zuul role
which means all zuul servers will get this dir. It doesn't currently
conflict with any of the cluster members' /var/lib/zuul contents so
should be fine.
Change-Id: I4709e3c7e542781a65ae24c1f05a32444026fd26
So we can stop/pull/start, move the pull tasks to their own files
and add a playbook that invokes them.
Change-Id: I4f351c1d28e5e4606e0a778e545a3a805525ac71
mod_mem_cache was removed in Apache 2.4 so all the bits of
configuration gated by the IfModule are currently irrelevant.
The replacement is socache, the in-memory version is "shmcb" (can also
hook up to memcache, etc.). Enable the socache module, and switch the
cache matching parts to use socache and then fall-back to disk cache
(this is what it says this will do in the manual [1])
The other part of this is to turn the CacheQuickHandler off. The
manual says about this [2]
In the default enabled configuration, the cache operates within the
quick handler phase. This phase short circuits the majority of
server processing, and represents the most performant mode of
operation for a typical server. The cache bolts onto the front of
the server, and the majority of server processing is avoided.
I won't claim to fully understand how our mod_rewrite rules and
mod_proxy all hang together with phases and what-not. But emperically
with this turned on (default) we do not seem to get any caching on the
tenant status pages, and with it turned off we do.
I've deliberately removed IfModule gating as well. This actually hid
the problem and made it much more difficult to diagnose; it is much
better if these directives just fail to start Apache if we do not have
the modules we expect to have.
[1] https://httpd.apache.org/docs/2.4/mod/mod_cache_socache.html
[2] https://httpd.apache.org/docs/2.4/mod/mod_cache.html#cachequickhandler
Change-Id: I4e5f803b9d4fb6c2351cf151a085b93a7fd20f60
These two values overwrite each other, move into common configuration.
The "cache-status" is a verbose string, so quote it.
Change-Id: I3cc4627de3d6a0de1adcfed6b424fc3ed0099245
We noticed that our zuul scheduler was running out of disk and one of
the causes of this is we are pulling all of the wonderful new zuul
images and not pruning them. This happens because we were only pruning
when (re)starting services and we don't do that automatically with Zuul.
Address this by always pruning after pulling even if we don't restart
services. This should be safe because prune will leave the latest tagged
images as well as the running images.
This should keep our disk consumption down.
Change-Id: Ibdd22ac42d86781f1e87c3d11e05fd8f99677167
We're running these in containers now. Please not to try to start
them the old way.
failed_when false is because we can't disable the old service
in the gate if there is no service file installed.
Change-Id: Ia4560f385fc98e23f987a67a1dfa60c3188816b6
We are starting to use the apache2 mod_cache_disk functionality more now
and during use the cache has grown into the 1.5GB range. The
htcacheclean process is cleaning up every 2 hours which is how it is
getting behind with its limit of 300MB. Reduce the interval to 15
minutes by supplying an /etc/default/apache-htcacheclean config.
Note we cache status.json files which are only valid for a very short
period of time. This likely explains the quick growth of the cache.
Change-Id: Iff00fb1806796ef6db26e53e026c533c47a902b4
If we need to start and stop, it's best to use playbooks.
We already have tasks files with start commands in each role,
so put the stop commands into similar task files.
Make the restart playbook import_playbook the stop and start
playbooks to reduce divergence.
Use the graceful shutdown pattern from the gerrit docker-compose
to stop the zuul scheduler.
Change-Id: Ia20124553821f4b41186bce6ba2bff6ca2333a99
Zuul is publishing lovely container images, so we should
go ahead and start using them.
We can't use containers for zuul-executor because of the
docker->bubblewrap->AFS issue, so install from pip there.
Don't start any of the containers by default, which should
let us safely roll this out and then do a rolling restart.
For things (like web or mergers) where it's safe to do so,
a followup change will swap the flag.
Change-Id: I37dcce3a67477ad3b2c36f2fd3657af18bc25c40
