openstack/openstack-ansible-ops
Code Issues Proposed changes
openstack-ansible-ops/bowling_ball
History
Nolan Brubaker dc46ab7ba6 Add Nova service test capability 
Basing Nova work directly on the OpenStackSDK so that there are fewer
(direct) dependencies in this code. Glance is only modified so that the
import naming is more explicit.

We don't create a VM in order to reduce dependencies on external
services such as neutron or cinder.

Change-Id: I5385754d29d5008ab5ea16bf84f4192051c5c70a
2017-05-31 12:33:15 -04:00
..
README.rst
Implement CLI test picker
2017-05-19 11:47:49 +00:00
rolling_restart.py
Unify output formatting, add logging
2017-03-15 16:36:32 -04:00
rolling_tests.py
Add Nova service test capability
2017-05-31 12:33:15 -04:00

README.rst

Bowling Ball - OpenStack-Ansible Rolling Downtime Simulator

date

2017-03-09

tags

rackspace, openstack, ansible

category

*openstack, *nix

About

This project aims to test for issues with rolling downtime on OpenStack-Ansible deployments. It's comprised of two main components:

  • The rolling_restart.py script
  • The rolling_test.py script

The rolling_restart.py script will stop containers from a specified group in a rolling fashion - node 1 will stop, then start, then node 2, then node 3 and so on. This script runs from the deployment host.

The tests directory contains scripts to generate traffic against the target services. system that will be restarted by rolling_restart.py in order to measure the effects. These scripts run from a utility container.

The rolling_test.py script contains tests to generate traffic against the target services. These vary per service, but attempt to apply usage to a system that will be restarted by rolling_restart.py in order to measure the effects. This script runs from a utility container.

Usage

  1. Start your test from a utility container. ./rolling_test.py keystone runs the Keystone test. ./rolling_test.py list will list tests and their descriptions

  2. From the deployment node, run rolling_restart.py in the playbooks directory (necessary to find the inventory script). Specify the service you're targeting with the -s parameter.

    rolling_restart.py -s keystone_container

    You can specify a wait time in seconds between stopping and starting individual nodes.

    rolling_restart.py -s keystone_container -w 60

Assumptions

These tools are currently coupled to OSA, and they assume paths to files as specified by the multi-node-aio scripts.

Container stopping and starting is done with an ansible command, and the physical host to target is derived from the current inventory.

rolling_restart.py must currently be run from the playbooks directory. This will be fixed later.

You must source openrc before running rolling_test.py.

Creating New Tests

Tests should subclass from the ServiceTest class in the same file and implement the following properties and methods:

  1. run - The actual test to run should be placed in this method. Timings

    will be gathered based on when this function starts and stops.

  2. pre_test - Any pre-test setup that needs to happen, like creating a

    file for Glance, Cinder, or Swift upload.

  3. post_test - Any post-test teardown that might be needed.
  4. service_name - The primary service that is being tested.
  5. description - Brief description of what the test does.

Finally, add the test to the available_tests dictionary with the invocation name as the key and the class as the value.

Why the name?

It sets 'em up and knocks em down.