From 780afe54e47d3d4acdf9e49013ec3bd4b4f84e88 Mon Sep 17 00:00:00 2001
From: Steve Wilkerson <wilkers.steve@gmail.com>
Date: Thu, 29 Jun 2017 10:26:01 -0500
Subject: [PATCH] Add docs detailing testing and gating requirements
Adds basic expectations and explanations of openstack-helms
testing and gating approaches. This will continue to be expanded
upon but serves to provide a base on which to build
Change-Id: I689446b7124c4e11a92c73ef04e95d1840c6dc0a
---
doc/source/gates.rst | 144 +++++++++++++++++++++++++++++++++++++++++
doc/source/index.rst | 2 +
doc/source/testing.rst | 61 +++++++++++++++++
3 files changed, 207 insertions(+)
create mode 100644 doc/source/gates.rst
create mode 100644 doc/source/testing.rst
diff --git a/doc/source/gates.rst b/doc/source/gates.rst
new file mode 100644
index 0000000000..343cdf51d9
--- /dev/null
+++ b/doc/source/gates.rst
@@ -0,0 +1,144 @@
+====================
+OpenStack-Helm Gates
+====================
+
+To facilitate ease of testing and debugging, information regarding gates and
+their functionality can be found here.
+
+OpenStack-Helm's single node and multinode gates leverage the kubeadm-aio
+environment created and maintained for use as a development environment. All
+information regarding the kubeadm-aio environment can be found here_.
+
+.. _here: https://github.com/openstack/openstack-helm/tree/master/tools/kubeadm-aio
+
+Gate Checks
+-----------
+
+OpenStack-Helm currently checks the following scenarios:
+
+- Testing any documentation changes and impacts
+- Running Make on each chart, which lints and packages the charts. This gate
+ does not stand up a Kubernetes cluster
+- Provisioning a single node cluster and deploying the OpenStack services. This
+ check is provided for: Ubuntu-1604, CentOS-7, and Fedora-25
+- Provisioning a multi-node Ubuntu-1604 cluster and deploying the OpenStack
+ services. This check is provided for both a two node cluster and a three
+ node cluster.
+
+
+Gate Functions
+--------------
+
+To provide reusable components for gate functionality, functions have been
+provided in the gates/funcs directory. These functions include:
+
+- Functions for common host preparation operations, found in common.sh
+- Functions for Helm specific operations, found in helm.sh. These functions
+ include: installing Helm, serving a Helm repository locally, linting and
+ building all Helm charts, running Helm tests on a release, installing the
+ helm template plugin, and running the helm template plugin against a chart.
+- Functions for Kubernetes specific operations, found in kube.sh. These
+ functions include: waiting for pods in a specific namespace to register as
+ ready, waiting for all nodes to register as ready, install the requirements
+ for the kubeadm-aio container used in the gates, building the kubeadm-aio
+ container, launching the kubeadm-aio container, and replacing the
+ kube-controller-manager with a specific image necessary for ceph functionality.
+- Functions for network specific operations, found in network.sh. These
+ functions include: creating a backup of the host's resolv.conf file before
+ deploying the kubeadm environments, restoring the original resolv.conf
+ settings, creating a backup of the host's /etc/hosts file before adding the
+ hosts interface and address, and restoring the original /etc/hosts file.
+- Functions for OpenStack specific operations, found in openstack.sh. These
+ functions include: waiting for a successful ping, and waiting for a booted
+ virtual machine's status to return as ACTIVE.
+
+Any additional functions required for testing new charts or improving the gate
+workflow should be placed in the appropriate location.
+
+
+Gate Output
+-----------
+
+To provide meaningful output from the gates, all information pertaining to the
+components of the cluster and workflow are output to the logs directory inside
+each gate. The contents of the log directory are as follows:
+
+- The dry-runs directory contains the rendered output of Helm dry-run installs
+ on each of the OpenStack service charts. This gives visibility into the
+ manifests created by the templates with the supplied values. When the dry-run
+ gate fails, the reason should be apparent in the dry-runs output. The logs
+ found here are helpful in identifying issues resulting from using helm-toolkit
+ functions incorrectly or other rendering issues with gotpl.
+- The k8s directory contains the logs and output of the Kubernetes objects. It
+ includes: pods, nodes, secrets, services, namespaces, configmaps, deployments,
+ daemonsets, and statefulsets. Descriptions for the state of all resources
+ during execution are found here, and this information can prove valuable when
+ debugging issues raised during a check. When a single node or multi-node
+ check fails, this is the first place to look. The logs found here are helpful
+ when the templates render correctly, but the services are not functioning
+ correctly, whether due to service configuration issues or issues with the
+ pods themselves.
+- The nodes directory contains information about the node the gate tests are
+ running on in openstack-infra. This includes: the network interfaces, the
+ contents of iptables, the host's resolv.conf, and the kernel IP routing table.
+ These logs can be helpful when trying to identify issues with host networking
+ or other issues at the node level.
+
+
+Adding Services
+---------------
+
+As charts for additional services are added to OpenStack-Helm, they should be
+included in the gates. Adding new services to the gates allows a chart
+developer and the review team to identify any potential issues associated with
+a new service. All services are currently launched in the gate via
+basic_launch.sh. Any new services are added to this file, and the required
+format for a basic service is:
+
+::
+
+ helm install --namespace=openstack ${WORK_DIR}/mistral --name=mistral
+
+Some services in the gate require specific overrides to the default values in
+the chart's values.yaml file. If a service requires multiple overrides to
+function in the gate, the service should include a separate values.yaml file
+placed in the tools/overrides/mvp directory. The <service>.yaml MVP files
+provide a configuration file to use for overriding default configuration values
+in the chart's values.yaml as an alternative to overriding individual values
+during installation. A chart that requires a MVP overrides file
+requires the following format:
+
+::
+
+ helm install --namespace=openstack ${WORK_DIR}/cinder --name=cinder \
+ --values=${WORK_DIR}/tools/overrides/mvp/cinder.yaml
+
+
+Adding Tests
+------------
+
+As new charts are developed and the services are added to the gate, an
+associated Helm test should be introduced to the gates. The appropriate place
+for executing these tests is in basic_launch.sh, and must be placed after the
+entry for installing the service and any associated overrides. Any tests that
+use the Rally testing framework should leverage the helm_test_deployment
+function in the aforementioned funcs/helm.sh file. For example, a Helm test for
+Mistral might look like:
+
+::
+
+ helm_test_deployment mistral 600
+
+This results in the gate running the following:
+
+::
+
+ helm test --timeout 600 mistral
+ mkdir -p mistral/rally
+ kubectl logs -n openstack mistral-rally-test > logs/rally/mistral
+ kubectl delete -n openstack pod mistral-rally-test
+
+Any tests that do not use the Rally testing framework would need to be handled
+in the appropriate manner in basic_launch.sh. This would ideally result in new
+functions that could be reused, or expansion of the gate scripts to include
+scenarios beyond basic service launches.
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 2991385a30..aad5e0e5f0 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -16,6 +16,8 @@ Contents:
devref/index
contributing
troubleshooting/index
+ testing
+ gates
Indices and tables
==================
diff --git a/doc/source/testing.rst b/doc/source/testing.rst
new file mode 100644
index 0000000000..8c82b9f639
--- /dev/null
+++ b/doc/source/testing.rst
@@ -0,0 +1,61 @@
+=======
+Testing
+=======
+
+Helm Tests
+----------
+
+Every OpenStack-Helm chart should include any required Helm tests necessary to
+provide a sanity check for the OpenStack service. Information on using the Helm
+testing framework can be found in the Helm repository_. Currently, the Rally
+testing framework is used to provide these checks for the core services. The
+Keystone Helm test template can be used as a reference, and can be found here_.
+
+.. _repository: https://github.com/kubernetes/helm/blob/master/docs/chart_tests.md
+
+.. _here: https://github.com/openstack/openstack-helm/blob/master/keystone/templates/pod-rally-test.yaml
+
+
+Testing Expectations
+--------------------
+
+Any templates for Helm tests submitted should follow the philosophies applied in
+the other templates. These include: use of overrides where appropriate, use of
+endpoint lookups and other common functionality in helm-toolkit, and mounting
+any required scripting templates via the configmap-bin template for the service
+chart. If Rally tests are not appropriate or adequate for a service chart, any
+additional tests should be documented appropriately and adhere to the same
+expectations.
+
+
+Running Tests
+-------------
+
+Any Helm tests associated with a chart can be run by executing:
+
+::
+
+ helm test <helm-release-name>
+
+The output of the Helm tests can be seen by looking at the logs of the pod
+created by the Helm tests. These logs can be viewed with:
+
+::
+
+ kubectl logs <test-pod-name> -n <namespace>
+
+Additional information on Helm tests for OpenStack-Helm and how to execute
+these tests locally via the scripts used in the gate can be found in the
+gates_ directory.
+
+.. _gates: https://github.com/openstack/openstack-helm/blob/master/tools/gate/
+
+
+Adding Tests
+------------
+
+All tests should be added to the gates during development, and are required for
+any new service charts prior to merging. All Helm tests are added to the gates
+in the basic_launch.sh_ file.
+
+.. _basic_launch.sh: https://github.com/openstack/openstack-helm/blob/master/tools/gate/basic_launch.sh