openstack/openstack-ansible-ops
Code Issues Proposed changes
openstack-ansible-ops/bowling_ball
History
Nolan Brubaker c2aeb03b10 Unify output formatting, add logging 
Since there is no central storage for the restart actions and the tests
at the moment, they should have timestamped output to correlate events.
Using a standard format will make this easier to do programmatically,
too.

The output is sent both to stdout and a log file. Additionally, Ansible
output is suppressed now, so it doesn't pollute the console. Log times
are in UTC, to avoid timezone mismatches between nodes (observed in
multi-node AIOs).

The `configure_logging` function was copied into the tests/keystone.py
file instead of put into a common module in order to keep that file a
standalone unit for the time being.

Change-Id: I399d1827a8559896ca87e45c00241293ccd033d2
2017-03-15 16:36:32 -04:00
..
tests
Unify output formatting, add logging
2017-03-15 16:36:32 -04:00
README.rst
Add rolling downtime simulation tools
2017-03-09 16:29:13 -05:00
rolling_restart.py
Unify output formatting, add logging
2017-03-15 16:36:32 -04:00

README.rst

Bowling Ball - OpenStack-Ansible Rolling Downtime Simulator

date

2017-03-09

tags

rackspace, openstack, ansible

category

*openstack, *nix

About

This project aims to test for issues with rolling downtime on OpenStack-Ansible deployments. It's comprised of two main components:

  • The rolling_restart.py script
  • The tests directory

The rolling_restart.py script will stop containers from a specified group in a rolling fashion - node 1 will stop, then start, then node 2, then node 3 and so on. This script runs from the deployment host.

The tests directory contains scripts to generate traffic against the target services. These vary per service, but attempt to apply usage to a system that will be restarted by rolling_restart.py in order to measure the effects. These scripts run from a utility container.

Usage

  1. Start your test script from the utility container. keystone.py will request a session and a list of projects on an infinite loop, for example.

  2. From the deployment node, run rolling_restart.py in the playbooks directory (necessary to find the inventory script). Specify the service you're targeting with the -s parameter.

    rolling_restart.py -s keystone_container

    You can specify a wait time in seconds between stopping and starting individual nodes.

    rolling_restart.py -s keystone_container -w 60

Assumptions

These tools are currently coupled to OSA, and they assume paths to files as specified by the multi-node-aio scripts.

Container stopping and starting is done with an ansible command, and the physical host to target is derivced from the current inventory.

rolling_restart.py must currently be run from the playbooks directory. This will be fixed later.

You must source openrc before running keystone.py.

Why the name?

It sets 'em up and knocks em down.