4efdd1cb1d
Boston was the first time we presented a section on the Project Status and Zuul pages. Some of the content was not well thought out or well organized. This patch adds some more details with additional screen shots that follow along the process of using the status page and better explain elastic rechecks. Change-Id: I69f5d3dc80645260e43a1e61625b5ad265d2d6b7
186 lines
4.8 KiB
ReStructuredText
186 lines
4.8 KiB
ReStructuredText
=================================
|
|
OpenStack Project Status and Zuul
|
|
=================================
|
|
|
|
.. image:: ./_assets/os_background.png
|
|
:class: fill
|
|
:width: 100%
|
|
|
|
.. note::
|
|
Tags: [new_dev] [dev]
|
|
|
|
OpenStack Infrastructure and Project Status
|
|
===========================================
|
|
- `http://status.openstack.org <http://status.openstack.org>`_
|
|
|
|
- Zuul
|
|
- Rechecks
|
|
- Reviews
|
|
- Bugday
|
|
- OpenStack-Health
|
|
|
|
.. note::
|
|
|
|
- Zuul is the project gating and automation system that tests and merges
|
|
changes as well as publishing releases and documentation.
|
|
- Rechecks has a list of bugs and associated information for
|
|
nondeterministic check/gate failures.
|
|
- Reviews has a list of important reviews based on blueprint and bug priority
|
|
organized by project.
|
|
- Bugday shows real-time stats during bug smash days.
|
|
- OpenStack-Health has a dashboard of overal OpenStack test results.
|
|
|
|
Zuul
|
|
====
|
|
|
|
.. image:: ./_assets/workflow-project-status-and-zuul-zuul-page.png
|
|
|
|
Zuul Pipelines
|
|
==============
|
|
- Check
|
|
|
|
- Run against all newly updated patch sets
|
|
- Runs unit testing, Pep8, docs/releasenote build and
|
|
functional tempest testing
|
|
- Zuul votes +1/-1 depending on test results
|
|
|
|
- Gate
|
|
|
|
- Only run after a patch is approved by a core reviewer
|
|
- More extensive testing than check pipeline
|
|
- Runs another unit test run along with additional tempest testing
|
|
- 'Gates' code entering the stable or master branches
|
|
|
|
Zuul Pipelines - cont.
|
|
======================
|
|
|
|
- Post
|
|
|
|
- Jobs run against a patch after it merges
|
|
- Documentation build/publishing, tarball generation, image build
|
|
|
|
.. note::
|
|
|
|
- The above jobs are examples of what is run in each pipeline.
|
|
What is actually run varies based upon the project being tested.
|
|
|
|
Exercise
|
|
========
|
|
- Look at the `Zuul <http://status.openstack.org/zuul>`_ status page
|
|
|
|
- Find the information that can be retrieved for each patch in a pipeline
|
|
|
|
- What happens if you click on a patch under test
|
|
|
|
- How many gate and check jobs are running
|
|
- Pick your favorite project and report how many jobs it has running in IRC
|
|
- What is the significance of the dots, lines and colors
|
|
|
|
- Discuss your findings with your group
|
|
|
|
Patch Number
|
|
============
|
|
|
|
.. image:: ./_assets/workflow-project-status-and-zuul-patch-id.png
|
|
:class: image-pad-top
|
|
|
|
.. note::
|
|
|
|
- The number next to 'Change' is the patch number.
|
|
- Can use the patch number to track status in Zuul status page.
|
|
|
|
Filtering on Patch Number in Zuul
|
|
=================================
|
|
|
|
.. image:: ./_assets/workflow-project-status-and-zuul-zuul-job-status.png
|
|
:class: image-pad-top
|
|
|
|
.. note::
|
|
|
|
- Shows the Zuul status page filtered down to just the patch of interest.
|
|
|
|
|
|
Zuul Failures
|
|
=============
|
|
|
|
.. image:: ./_assets/workflow-project-status-and-zuul-zuul-fail.png
|
|
:class: image-pad-top
|
|
|
|
|
|
|
|
Analysing Zuul Failures
|
|
=======================
|
|
- Logs may be accessed by clicking on the test's name
|
|
|
|
- Directs the user to saved log output
|
|
- Uses the browser to view the logs
|
|
|
|
- Voting and non-voting jobs
|
|
|
|
- Voting job failures cause a -1 vote from Zuul on the patch
|
|
- Non-voting jobs do not cause a -1 vote from Zuul upon failure
|
|
|
|
- Non-voting jobs are new jobs that are being tested and may not yet be ready
|
|
to vote
|
|
|
|
Logs
|
|
====
|
|
|
|
.. image:: ./_assets/workflow-project-status-and-zuul-zuul-failure-log-list.png
|
|
:class: image-pad-top
|
|
|
|
.. note::
|
|
|
|
- Example of what the logs for a run look like
|
|
- Actual log files will vary depending on the test you are examining
|
|
|
|
Logs - Testr HTML Report
|
|
========================
|
|
|
|
.. image:: ./_assets/workflow-project-status-and-zuul-failure-testr-html.png
|
|
:class: image-pad-top
|
|
|
|
|
|
Logs - Job Run Output
|
|
=====================
|
|
|
|
.. image:: ./_assets/workflow-project-status-and-zuul-failure-log.png
|
|
:class: image-pad-top
|
|
|
|
Intermittent Failures
|
|
=====================
|
|
- Caused by timing/connectivity issues in check/gate
|
|
- May receive elastic recheck message
|
|
- List of `Rechecks <http://status.openstack.org/elastic-recheck>`_
|
|
- To trigger a re-run of check or gate add a comment to the patch
|
|
in the form of 'recheck bug XXXXX'
|
|
|
|
Recheck Example
|
|
===============
|
|
|
|
.. image:: ./_assets/workflow-project-status-and-zuul-recheck-example.png
|
|
:class: image-pad-top
|
|
|
|
.. note::
|
|
|
|
- Note ability to see logstash and link to launchpad
|
|
|
|
What Are Rechecks
|
|
=================
|
|
|
|
- Issues causing intermittent failures
|
|
- Elastic search is used to look for logs indicating similar failures
|
|
|
|
- Users add new bugs to
|
|
`elastic-recheck <https://docs.openstack.org/infra/elastic-recheck/readme.html>`_
|
|
|
|
- Goal is to capture all instances of a failure in the hopes of identifying
|
|
patterns causing the bug
|
|
- Comments on patches that hit a known bug
|
|
|
|
Exercise
|
|
========
|
|
- Find how rechecks are categorized
|
|
- Discuss with your table how you would determine you encountered
|
|
one of these bugs
|