Without this our config changes are not applying to the running service
until something else reloads or restarts the service.
Change-Id: I4df229d1c42f06159a4b320d4b6a07c5239ca111
This makes the haproxy role more generic so we can run another (or
potentially even more) haproxy instance(s) to manage other services.
The config file is moved to a variable for the haproxy role. The
gitea specific config is then installed for the gitea-lb service by a
new gitea-lb role.
statsd reporting is made optional with an argument. This
enables/disables the service in the docker compose.
Role documenation is updated.
Needed-By: https://review.opendev.org/678159
Change-Id: I3506ebbed9dda17d910001e71b17a865eba4225d
The usptream haproxy image switched to running as a user, rather than
as root. This means it can not bind to 80/443 and instantly dies.
I've added a comment with some discussion, but for now, use the root
user.
[1] 82ff028a25
Change-Id: Ic9b04cdd09f73d9df015bcb173871cff1ae58835
The haproxy 2.4 images aren't working for us, docker-compose
perpetually reports the container in a "restarting" state. Pin back
from latest to 2.3 until we can sort out what needs to change in how
we integrate this on the server.
Change-Id: I01ae11a31eb8eaeb9e570692d5ec268395f69a97
This moves the haproxy containers to syslog with tags that will get
each container's output filtered into /var/log/containers.
Change-Id: I65294baec3c092ede5ee97856d8d879174b0d8d4
Adding the tcplog option to an haproxy backend definition overrides
the default log format. Remove it so the supplied default (which we
based on the tcplog built-in default with some additions) will be
used instead.
Change-Id: Id302dede950c1c2ab8e74a662cc3cb1186a6593d
When forwarding TCP sockets at OSI layer 4 with haproxy, it helps to
know the ephemeral port from which it sources each connection to the
backend. In this way, backend connections can be mapped to actual
client IP addresses by correlating backend service access logs with
haproxy logs.
Add "[%bi]:%bp" between the frontend name and backend name values
for the default log-format documented here:
https://www.haproxy.com/blog/haproxy-log-customization/
Change-Id: Ic2623d483d98cd686a85d40bc4f2e8577fb9087f
The increase in connection volume is not sustainable for the available memory on the backend servers. We'll likely need to scale the cluster before reattempting this.
This reverts commit 79f363164ed0c81e4c7603885f8e9815164b2df2.
Change-Id: Ibe64f472633a62df659c6183aa96e095dda7fdbc
We've set maxconn to 4k concurrent connections on the front side of our
haproxy load balancer. Currently that seems to be creating a large
backlog of requests. Looking at cacti it appears that we have maybe up
to ~6-8 times this amount of overhead in resources on the gitea
backends. Be a little conservative and bump this value up by 4x and tune
from there.
Change-Id: I56d43b52c23f251cc632315c3b57e45541722970
We want to use stop_grace_period to manage gerrit service stops. This
feature was added in docker-compose 1.10 but the distro provides 1.5.
Work around this by installing docker-compose from pypi.
This seems like a useful feature and we want to manage docker-compose
the same way globally so move docker-compose installation into the
install-docker role.
New docker-compose has slightly different output that we must check for
in the gitea start/stop machinery. We also need to check for different
container name formatting in our test cases. We should pause here and
consider if this has any upgrade implications for our existing services.
Change-Id: Ia8249a2b84a2ef167ee4ffd66d7a7e7cff8e21fb
To make it clear that docker hub is but one of many possible registries,
update our usage of FROM and image: lines to include docker.io in the
path.
There are a few other FROM lines for the gitea images which are handled
in a separate stack.
Change-Id: I6fafd5f659ad19de6951574afc9a6b6a4cf184df
Add an Ansible handler to send a hangup signal through
docker-compose to the running haproxy daemon any time the task to
update its configuration fires.
Change-Id: I1946c1e7eaaa8a8e2209007b5d065dba952ec6e2
This adds the simplest form of health checking to haproxy, a tcp check
to the backends. We can do more sophisticated checks like checking ssl
negotiates or even HTTP requests but for now this is probably a good
improvement.
Change-Id: I3c6b07df4b3e0c380c757e1e5cb51ae0be655f34
Haproxy wants to log to syslog (and not stdout for performance reasons,
see https://github.com/dockerfile/haproxy/issues/3). However there is no
running syslog in our haproxy container. What we can do is mount in the
host's /dev/log and have haproxy write to the hosts syslog to get
logging.
Do this via a docker compose volume bind mount.
Change-Id: Icf4a91c2bc5f5dbb0bfb9d36e7ec0210c6dc4e90
We were using the leastconn method which sends new connections to the
backend with the least number of connections. Unfortunately git clients
seem to have trouble with varying backend repo state (due to GC and
packing) and the thought is sending all requests from a single client to
a single backend will alleviate this.
To do this we switch to the source balance method which hashes the
source IP and finds a stable backend to talk to. This method handles
backend outages fine as it will hash to a new backend if the older one
goes offline.
Change-Id: I2c7a4ec0809a2f4ef6556833ac6a0ff3651904dd
Build a container image with the haproxy-statsd script, and run that
along with the haproxy container.
Change-Id: I18be70d339df613bf9a72e115e80a6da876111e0
Socat is useful for managing haproxy through the haproxy management
socket. Install it when we install haproxy.
Change-Id: Ie2b16cef62f661669756d24d4a69ac1683401268
This ensures that we cleanup images that are superceded and no longer
necessary. We do this to avoid filling the disk with docker images.
Note that we use the -f flag to avoid being prompted by docker image
prune for confirmation.
Change-Id: I8eb5bb97d8c66755e695498707220c9e6e7b2de0
This will make sure that the latest relevant images are in the
local image storage, and therefore, will cause docker-compose up
to recreate containers when the images are updated.
Change-Id: Ic6f0bc8c8aea5b5c16501f4ab5d3095fb70c0ff7
This runs an haproxy which is strikingly similar to the one we
currently run for git.openstack.org, but it is run in a docker
container.
Change-Id: I647ae8c02eb2cd4f3db2b203d61a181f7eb632d2
