After I replace the docker packages some services I thought would come
back did not.
Lodegit seems to be an oversight, add restart always.
Also make sure the ZK containers start themselves.
I believe with Gerrit we've made the choice to not start automatically
due to the general high-touch nature of restarts. Keep the database
consistent and remove the auto restart there.
Change-Id: I98fa3055ac269564ed96570df0700b2aad24e4d2
This adds a program, zookeeper-statsd, which monitors zookeeper
metrics and reports them to statsd. It also adds a container to
run that program. And it runs the container on each of the
ZooKeeper quorum members. And it updates the graphite host to
allow statsd traffic from quorum members. And it updates the
4-letter-word whitelist to allow the mntr command (which is used
to gather metrics) to be issued.
Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
To prepare for switching to TLS, set up TLS certs for Zookeeper and
all of Nodepool and Zuul, but do not have them connect over TLS yet.
We have observed problems with Kazoo using TLS in production. This
will let us run the ZK quorum using TLS internally, and have Zuul
and Nodepool connect over plaintext while also exposing the TLS
client port so that we can perform some more production tests.
Change-Id: If93b27f5b55be42be1cf6ee23258127fab5ce9ea
This reverts commit 29825ac18b58145f007f64b2998357445b8fdd91.
We observed this issue in production:
https://github.com/python-zk/kazoo/issues/587
Revert until we find a fix.
Change-Id: Ib7b8e3b06770a83b39458d09d2b1e655bd94bd22
This creates TLS certs for Zookeeper, uses them inside the ZK
quorum, and configures Nodepool and Zuul to use them as well.
A full system restart of all ZK-related components will be required
after merging this patch.
Change-Id: I0cb96a989f3d2c7e0563ce8899f2a5945ea225b3
Migration plan:
* add zk* to emergency
* copy data files on each node to a safe place for DR backup
* make a json data backup: zk-shell localhost:2181 --run-once 'mirror / json://!tmp!zookeeper-backup.json/'
* manually run a modified playbook to set up the docker infra without starting containers
* rolling restart; for each node:
* stop zk
* split data and log files and move them to new locations
* remove zk packages
* start zk containers
* remove from emergency; land this change.
Change-Id: Ic06c9cf9604402aa8eb4bb79238021c14c5d9563
