741f5b333d
There are two issues in the zuul merger and executor shutdowns. The first is that `docker-compose ps -q` will report exited containers unlike `docker ps -q`. This means we may try to exec into a non running container which is an error. Handle this by checking the error message and proceeding if the 'is not running' string is present. The second issue is a race between stopping a container and running an exec in the container. If a container stops while an exec is running in it that exec appears to be treated with some equivalent of kill -9. The result is the exec returns 137. While theoretically possible for both executor and merger graceful stop command we seem to only hit this with the merger so we handle exit code 137 for the merger only. This way we'll get info if the executors start running into this too. Change-Id: Ia6dc2d7e397631d72968ffa89c4492b803c89c47
36 lines
1.2 KiB
YAML
36 lines
1.2 KiB
YAML
- name: Check if Zuul Executor containers are running
|
|
# It is possible they are stopped due to some external circumstance.
|
|
# NOTE: docker-compose ps -q reports exited containers unlike docker ps -q
|
|
command:
|
|
cmd: docker-compose ps -q
|
|
chdir: /etc/zuul-executor
|
|
become: true
|
|
become_user: root
|
|
register: executor_container_list
|
|
- name: Gracefully stop Zuul Executor
|
|
shell:
|
|
cmd: docker-compose exec executor zuul-executor graceful
|
|
chdir: /etc/zuul-executor
|
|
become: true
|
|
become_user: root
|
|
# Only run the docker exec command if a container is running
|
|
when: executor_container_list.stdout_lines | length > 0
|
|
register: ze_graceful
|
|
failed_when:
|
|
- ze_graceful.rc != 0
|
|
# If the exec fails because the container is not running we continue.
|
|
- "'is not running' not in ze_graceful.stderr"
|
|
- name: Wait for Zuul Executor to stop
|
|
shell:
|
|
cmd: docker-compose ps -q | xargs docker wait
|
|
chdir: /etc/zuul-executor
|
|
become: true
|
|
become_user: root
|
|
when: executor_container_list.stdout_lines | length > 0
|
|
- name: Remove Zuul Executor containers
|
|
shell:
|
|
cmd: docker-compose down
|
|
chdir: /etc/zuul-executor
|
|
become: true
|
|
become_user: root
|