116 Commits

Author SHA1 Message Date
Jeremy Stanley
31c531b250 Make subunit2html.py Py3k compatible.
* modules/jenkins/files/slave_scripts/subunit2html.py: Since some
projects call this script from under tox, in a py33 environment it
won't work as intended. Correct multiple instances of print as a
statement rather than as a function.

Change-Id: I179d6aee976b033a45f483fd92f058a6079f8f46
2013-07-30 18:31:54 +00:00
Jenkins
735e4114af Merge "configure the jenkins logger" 2013-07-24 14:10:37 +00:00
Khai Do
4f46000db2 fix upload to jenkins-ci bug.
The '-u' option was making curl prompt for a password,
which we don't want.  Remove the '-u' option to
automatically pass in jenkins user credentials to curl
via the config file.

Change-Id: I5766bcfef561e6ac10b49c958394d9601e62e1b5
2013-07-23 15:44:05 -07:00
Khai Do
ac8142f6ee configure the jenkins logger
This commit adds a logging configuration file for jenkins so
that we can persist the jenkins logger configuration across
restarts.

This commit also adds a logger configuration,
"org.gearman.session.logger" to "WARNING"

Change-Id: I4cf886a93f33bd153a6e3f81534df582425bd8f5
2013-07-23 11:59:46 -07:00
Jenkins
ac1a4b44c6 Merge "Use ssl for PyPI uploads" 2013-07-22 22:41:19 +00:00
Jeremy Stanley
edfc75e027 Clear workspace before cloning in GGP.
* modules/jenkins/files/slave_scripts/gerrit-git-prep.sh: The
git-clone subcommand fails spectacularly when its destination is set
to a non-empty directory. In the case of gerrit-git-prep this can
happen when files are left behind in a workspace with no .git tree,
and then the job is changed to run GGP later. Work around it by
forcibly removing any previous workspace contents before cloning,
but show a listing of any contents first for the benefit of
troubleshooting such situations.

Change-Id: I0662ee68ee2a0e7a66a4d85c8cc7aea8bf0c17c0
2013-07-15 22:54:32 +00:00
zaro0508
9e258ea5aa workaround for jenkins bug https://issues.jenkins-ci.org/browse/JENKINS-14193
The jenkins maven builder cannot reference environment variables if it is set
by the EnvInject plugin at the prebuilder step.  However the maven builder CAN
reference variables when loaded from the build environment setup (wrapper) step.
The only problem is that gerrit-git-prep only runs in the prebuilder step
which occurs aftern a build environment step (wrapper).  We want the
maven-properties.sh script to run after gerrit-git-prep because the
it queries for git info. A workaround is to call gerrit-git-prep
inside of the maven-properties script so it will run in the correct order.

Change-Id: Ie0a72d8509f5f1453e027486285c2817dbc67eb5
Reviewed-on: https://review.openstack.org/36879
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-07-12 23:22:14 +00:00
Mark McLoughlin
71726d834a Only skip openstack tarballs in requirements check
We really only want to allow openstack tarball requirements to be added
without checking, not just any tarball.

Change-Id: I0ce458c95dc2fba07100b921d094ff174ed7da57
Reviewed-on: https://review.openstack.org/36070
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-07-12 23:08:59 +00:00
Monty Taylor
7937ccda42 Use ssl for PyPI uploads
Change-Id: I7f371133931d142b0141a3ca7e35558731a81e79
2013-07-12 18:32:53 -04:00
Clark Boylan
733fd110e4 Revert "Add git fetch --tags to gerrit-git-prep.sh."
This reverts commit dbe5f35a5a595a982316cfffa978b572c17ada5e.

Git fetch operations are time consuming on our Centos6 slaves. The
additional fetch for the tags is nice to have but not necessary in most
cases. Remove it to reduce time spent doing this on the Centos6 slaves.

Change-Id: If2ca089df660b6dda1fb6381a8ea5a2ef6e5304f
Reviewed-on: https://review.openstack.org/36544
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-07-10 21:54:50 +00:00
Mark McLoughlin
3d6e24418a Use dev-requirements.txt in openstack/requirements
We wish to be able to use pre-release versions of some libraries during
the development cycle. The principle is that we don't want to expose
users of the stable release of those libraries to the churn that happens
during a development cycle. More background on this is described here:

  https://wiki.openstack.org/wiki/Oslo

The requirements.txt files in this repository are used for two purposes.
Firstly to describe what libraries projects are allowed to use and,
secondly, what libraries to populate our PyPI mirror with.

In the case of these development versions of libraries, they aren't
released to PyPI so we don't want them to be appear in our PyPI mirror.
So, we need to add a file which is only used for the first purpose and
ignored by the mirror-building job.

We allow dev-requirements.txt to list newer versions of libraries which
are already listed in requirements.txt so that we still get the stable
version in our mirror.

Change-Id: I176d40404adac6f7dcb2a255b9c42eb3d2c9321e
Reviewed-on: https://review.openstack.org/36128
Reviewed-by: Doug Hellmann <doug.hellmann@dreamhost.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Julien Danjou <julien@danjou.info>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-07-10 19:09:57 +00:00
zaro0508
d1478aabf7 run script and load properties in correct order
By default the EnvInject plugin loads the properties
file before running the bash script when defined in wrapper
section.  However we want the opposite execution order, run script
then load properties file.

To work around this ordering problem we need to define
two instances of EnvInject, 1st instance to run the
script and 2nd instance to load the properties file.
This cannot be done in jenkins UI, but apparently works
when defined in JJB.

This will also let us run gerrit-git-prep in the
more conventional pre-builder step.

Change-Id: I7a16143a2f9b79d2dbe2684a7d016f6640f5a707
Reviewed-on: https://review.openstack.org/34936
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: matthew wagoner <zxkuqyb@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-07-08 22:25:06 +00:00
daisy-ycguo
94b59469d8 Improve the robustness of manuals translation scripts
Add codes to get the list of document's names in openstack manuals,
other than hard code.
Check whether the files/folders exist before taking actions,
to improve the robustness.

bug 1195180

Change-Id: I3e3d17a77920c99bed6881ac0cc460b1fce54cea
Reviewed-on: https://review.openstack.org/34700
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-07-08 22:15:54 +00:00
Clark Boylan
dbe5f35a5a Add git fetch --tags to gerrit-git-prep.sh.
* modules/jenkins/files/slave_scripts/gerrit-git-prep.sh: Run
`git fetch --tags origin` to make sure that we have all available
tags`.

Change-Id: I2cfd691659cd0e10f5295fbbe66b36b818d6d6b6
Reviewed-on: https://review.openstack.org/35527
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-07-03 19:52:31 +00:00
Mark McLoughlin
e13eab706a Ignore --find-links lines in requirement checks
Fixes bug #1194807

In quantum, we currently have a URL based requirement:

  http://.../oslo.config-1.2.0a2.tar.gz#egg=oslo.config-1.2.0a2

The requirements check currently ignores this.

It turns out that pip has a bug which doesn't where you can end up with
the oslo.config 1.1.1 code installed. This is because oslo.config>=1.1.0
gets pulled in as a transitive dep and pip gets confused. You can
reproduce with e.g.

  $> pip install \
       http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  ImportError: cannot import name DeprecatedOpt

This is because of a bug with pip where it sees oslo.config-1.2.0a3 and
oslo.config as two unrelated things. It should strip the version part of
the egg= fragment before using it as a package name, but it doesn't.

However, we can simply use the -f/--find-links pip option in our
requirements.txt to add the tarball URL to the list of URLs considered
and also add the oslo.config>=1.2.0a3 dependency:

  $> pip install \
       -f http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       'oslo.config>=1.2.0a3' \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'

This is actually exactly the semantics we want and we go to great
lengths in pbr to get these semantics while using a single tarball URL.
The only downside to this --find-links strategy is that we gain an extra
line in our requirements.txt ... but it does work around the pip bug.

I think it makes sense for the requirements check to just ignore
--find-links lines for now like it does for URLs and -editable lines.
Using this method means we actually do require new versions of libraries
consumed this way to be approved into openstack/requirements first since
we have an explicit 'oslo.config>=1.2.0a3' listed rather than that being
derived from a URL.

It may make sense in future to have automation around checking which
find-links URLs are allowed ... but the same can be true for normal
dependency URLs. This change allows us to move forward and use latest
oslo.config in Nova, Neutron, etc. without falling foul of the pip bug.

Change-Id: I6f3eb5fd2c75615d9a1cae172aed859b36b27d4c
Reviewed-on: https://review.openstack.org/35296
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
2013-07-02 23:14:03 +00:00
zaro0508
3585081d4d workaround for jenkins bug https://issues.jenkins-ci.org/browse/JENKINS-14193
The jenkins maven builder cannot reference environment variables if it is set
by the EnvInject plugin at the prebuilder step.  However the maven builder CAN
reference variables when loaded from the build environment setup (wrapper) step.
The only problem is that gerrit-git-prep runs in prebuilder step which occurs after
a build environment step.  We want the script to run after gerrit-git-prep so
we are calling the gerrit-git-prep in script so it will run in correct order.

Change-Id: Iedd7561b4ef8e19b3713b63bb76b9b754ce116e9
Reviewed-on: https://review.openstack.org/34023
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-06-25 21:35:25 +00:00
daisy-ycguo
46c3eadc7c Fixed the issue: missed pot files cause translation scripts error.
Missed pot files will cause an error in manuals translation scripts.
So move the "tx set --auto-local" command backwards
after the command to generate pot files.

bug 1193127

Change-Id: I57553a9b295796a70eaf8b54e2c9037f928cdc06
Reviewed-on: https://review.openstack.org/33909
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-06-25 20:57:02 +00:00
Monty Taylor
358da22571 Escape newlines and " in metadata script.
Change-Id: If0c8667042fe9f630ac04f1ca5c7f390fdcbc94a
Reviewed-on: https://review.openstack.org/34058
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-06-21 23:16:41 +00:00
Clark Boylan
5fbf5dd6c1 Use ZUUL_ prefix instead of GERRIT_ prefix in jobs
* modules/jenkins/files/slave_scripts/create-ppa-package.sh:
* modules/jenkins/files/slave_scripts/docbook-properties.sh: Use
ZUUL_REFNAME instead of GERRIT_REFNAME as the jenkins gearman plugin
does not receive GERRIT_* parameters.

Change-Id: I5869b277e9344b8805997a3c702e1ae3af0a9e8f
Reviewed-on: https://review.openstack.org/34052
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-21 21:37:59 +00:00
zaro0508
39019818f6 Fix setting of properties
running the bash script with a properties build wrapper doesn't work
because the wrapper runs before a scm checkout since the
script queries git it needs to be run after checkout.  This
commit uses the shell builder to run the script then the
evninject builder to read the generated properties file
to make it available to the rest of the build.

Fixed the script to always return 0 exit code on git commands.
Also changed $() to back ticks for consistency.

Change-Id: I1f31076762fb1d33d7e889fd2de4cf28d64d922d
Reviewed-on: https://review.openstack.org/33471
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-06-19 21:16:21 +00:00
zaro0508
782501b228 make maven versioning and deployment workflow similiar to python
This commit fixes bug 1182154

The maven workflow for versioning and deployment is at odds with how
we like to do it.  For versioning, our convention is to get info
from SCM to set the project build versions.  For deployments
to remote repositories we prefer to use curl instead of mvn deploy
due to maven security vulnerabilities.  Our python builds
have already been setup to set package versions from SCM and
deploy to pypi using curl so this commit is to make
maven versioning and deployments similiar our python
versioning and deployment workflow.

This commit does the following:

Setup a project version string as an environement variable
so we can pass it to maven builds. The version string is
retrieved from information in git.  This makes the build versioning
workflow similar to how we build python packages.

  This setup expects a variable called '$project-version' in
  the root 'version' element (i.e. <version>${project-version}</version>)
  of the maven project's pom.xml file.

  For general (throw away CI) builds we do the following:
    1. generate a package 'myplugin.hpi' with version '1.3.0.4.a0bc21f'
       in the MANIFEST.MF file.  The '4' is the number of commits since
       last tag.
    2. publish 'myplugin-1.3.0.4.a0bc21f.hpi' to tarballs.o.o

  For release builds we do the following:
    1. generate a package 'myplugin.hpi' with version '1.3.1'
     in the MANIFEST.MF file.
    2. publish 'myplugin-1.3.1.hpi' to tarballs.o.o
    3. publish 'myplugin-1.3.1.hpi' to repo.jenkins-ci.org

Passes the jenkins credentials from hiera to the pypi slave so
we can use it to deploy released plugins to repo.jenkins-ci.org

Creates a generic jenkinsci-upload job that will deploy
released jenkins plugin artifacts to a remote repository with
user credentials from hiera. It will use the same curl deployment
method as the pypi-upload job.

Change-Id: If1306523a28da94ee970d96b7a788ca116348de7
Reviewed-on: https://review.openstack.org/31875
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-17 23:52:36 +00:00
daisy-ycguo
8e35c3117a Disable fuzzy matching in transifex automation scripts
Fuzzy matching can cause unexpected problems.
So add option "--no-fuzzy-matching" to command "python setup.py update_catalog"
in propose_translation_update and upstream_translation_update scripts.

Bug 1176164

Change-Id: I632d6eb9280ca11883760f17fc4363275692b38b
Reviewed-on: https://review.openstack.org/31472
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-10 21:17:51 +00:00
James E. Blair
4e5201f48b Fix run-docs as a gate job.
If run-docs is running as a gate script, ZUUL_REFNAME is not set.
We also don't care where the docs end up so do nothing (treat as
master).

Change-Id: I8f5a7ba4c6ab714765871c60b5e587fcb4dece41
Reviewed-on: https://review.openstack.org/31889
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-06 00:14:47 +00:00
James E. Blair
c108532b32 Don't publish non-master docs to master location.
The current run-docs script only moves the publishing location on
tags or stable branches.  This makes it so that master docs stay
where they are, stable branches and tags have the current behavior,
and anything else is placed in the branch name.  This should affect
milestone-proposed (which will be published to .../milestone-proposed/)
and feature branches (.../feature/foo).  Currently, these two cases
are in a race with master -- last change wins.

Change-Id: If1e09d31a7001255746ba1a91b641b95f42fcfbb
Reviewed-on: https://review.openstack.org/31300
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-06-05 22:28:44 +00:00
Clark Boylan
ff91e70e42 set -e when checking testr test counts.
* modules/jenkins/files/slave_scripts/run-tox.sh: set -e when checking
test counts so that the case when no tests are found and test complains
about a missing integer expression we fail the Jenkins job.

Change-Id: I2b416637d02d0b87fd3f9ece323e168c039da38b
Reviewed-on: https://review.openstack.org/30818
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
2013-05-31 11:57:13 +00:00
Monty Taylor
40c9a7a405 Don't suppress errors in run-cover.
Seriously. Why did I think that was a good idea?

Change-Id: I161a97d36592f271c7f2b1291acfd87b12199118
Reviewed-on: https://review.openstack.org/30524
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-26 03:03:18 +00:00
Monty Taylor
d2de309f72 Port run-tox.sh change in to run-cover.sh
cover can run on CentOS nodes, so we need this here too.

Change-Id: I6bc4a69559ecd90f12700910c1438962690e9e54
Reviewed-on: https://review.openstack.org/29685
Reviewed-by: matthew wagoner <zxkuqyb@gmail.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-20 18:26:37 +00:00
Jeremy Stanley
0ae2078416 CentOS/RH support for jenkins-sudo-grep.sh script.
* modules/jenkins/files/slave_scripts/jenkins-sudo-grep.sh: The log
file names and sudo error patterns are different in RHEL and CentOS
than in Debian and Ubuntu, so this leverages puppet's facter utility
to determine which OS family is in use and look in the correct place
for the correct error message.

Change-Id: Iaf3bbc60e408bac575d3724fb6181a5976c8004f
Reviewed-on: https://review.openstack.org/28649
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-10 21:48:10 +00:00
daisy-ycguo
dc21d725b2 Add script to upload/download translation resources of manuals.
Add two new translation jobs for manulas translation uploading and downloading

Bug 1162701

Change-Id: I2a07d588a85fad1c780984b3635b0795495e56f2
Reviewed-on: https://review.openstack.org/25810
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Paul Belanger <paul.belanger@polybeacon.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: matthew-wagoner <matthew.wagoner@hp.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-03 18:40:27 +00:00
annegentle
8e113d46c0 Splits the basic-install guide into two builds for Fedora/Ubuntu.
Change-Id: Ic3697239f2ae0421bfe3faab4dcd1450602deb8a
Reviewed-on: https://review.openstack.org/27443
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Anita Kuno <anita.kuno@enovance.com>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-29 20:16:15 +00:00
Jeremy Stanley
55123aa1d3 Debugging output for docbook properties script.
* .../jenkins/files/slave_scripts/docbook-properties.sh: Have the
shell echo commands run in this script, to aid in debugging. The
console log for jobs using this didn't previously provide sufficient
feedback to be able to see what values were set in the resulting
properties file.

Change-Id: I9909009d3f85e6c3c1e5dbc5465ef38adbd75df4
Reviewed-on: https://review.openstack.org/26849
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-12 22:41:40 +00:00
Clark Boylan
e3852213b0 Make subunit2html.py python2.6 compatible.
* modules/jenkins/files/slave_scripts/subunit2html.py: Swap out unittest
with testtools for python 2.6 compatibility. Remove argparse and
manually parse argv for python 2.6 compatibility.

Change-Id: If3a253439550b5394b9b7eac129d8ed66b7ba951
Reviewed-on: https://review.openstack.org/26785
Reviewed-by: lifeless <robertc@robertcollins.net>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
2013-04-12 14:49:25 +00:00
James E. Blair
05cd301f8b Clone requirements into a tempdir.
Git doesn't remove git repos with git clean, so the workspace
checkout of the requirements repo was stale.  Clone into a
tempdir instead.

Also, fix a traceback in the case that a new requirement is added.
The loop wasn't short-circuiting as it should in that case.

Print the git sha from the requirements repo to aid in debugging.

Reorder imports.

Change-Id: Id5a8e748a1fb353f2628a9c9fa291d6825d1046d
Reviewed-on: https://review.openstack.org/26791
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-04-11 21:16:28 +00:00
Robert Collins
c8962dea27 Automatically handle subunit v1 or v2 streams.
Change-Id: Ib1ef05a6aa66606d03eca81d50ba84f6e6f41c1b
Reviewed-on: https://review.openstack.org/26688
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-04-11 21:10:51 +00:00
Clark Boylan
da9eba6e53 When running testr outside of venv use full path.
* modules/jenkins/files/slave_scripts/run-tox.sh: When running testr
from outside of the venv provide a full path to testr. Testr cannot be
found otherwise.

Change-Id: I25602847d389d4ddc62acd872e54727d6f309443
Reviewed-on: https://review.openstack.org/26783
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-04-11 20:03:38 +00:00
Clark Boylan
0466433260 Check tests run are not less than tests found.
* modules/jenkins/files/slave_scripts/run-tox.sh: With new subunit it is
possible for the number of tests run to be greater than the number of
tests found. Change the test count check from an inequality to check to
a ran less than found check to accomodate this.

Change-Id: Ibd109196ab23c57a8bb728e8a845027cd4876801
Reviewed-on: https://review.openstack.org/26645
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-04-10 22:53:52 +00:00
Clark Boylan
2a8e6841a3 Handle new subunit version 2 log files.
* modules/jenkins/files/slave_scripts/run-tox.sh: Subunit v2 log files
written by testr will output to .testrepository/$id.2. If this file is
present copy it instead of .testrepository/$id as the subunit log.

Change-Id: I3b43aa2c267773ee9d51ca369f0236e8bf9d628d
Reviewed-on: https://review.openstack.org/26417
Reviewed-by: lifeless <robertc@robertcollins.net>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-04-10 22:53:49 +00:00
Monty Taylor
1d2db8a09e Work around tox+rhel+distribute+setup_requires.
Trying to use setup_requires with the version of distribute on rhel
doesn't work because that version doesn't properly re-import itself.
When tox gets involved, it tries to run setup.py sdist outside of
the created virtualenv, which means it's ignoring our declared deps
for that action. The bug in distribute is worked around by running
any setup.py command once and letting it fail, then running again.

So - python setup.py --version is the cheapest setup.py command to
trigger the behavior that needs to happen. Throw the error results to
/dev/null because we expect there to be a traceback and we don't
want to confuse people reading log outputs. Leave stdout because
it's good to be able to track which setup_requires requirements
it is pulling down before it dies on not actually being able to
use them.

Change-Id: I982a08ffa5784ed97f70f41d91a58da66b7498b6
Reviewed-on: https://review.openstack.org/26533
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-09 22:44:30 +00:00
Clark Boylan
a0792fddc0 Make subunit2html compatible with subunit>=0.0.11.
* modules/jenkins/files/slave_scripts/subunit2html.py: Python subunit
ByteStreamToStreamResult takes an argument of non_subunit_name and not
non_subunit_input. Correct the argument name. Use argparse to parse the
command line arguments and add a new -2 options that specifies subunit
version 2 format logs should be read in. Otherwise read in version 1
subunit format.

This commit makes the second file name arg to subunit2html a required
argument and it is no longer optional. run-tox.sh already supplies both
file name arguments so this shouldn't break Jenkins.

Change-Id: I90804d4d12d77a544451c1647a137dc4b3b38f17
Reviewed-on: https://review.openstack.org/26430
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-04-09 16:37:14 +00:00
Clark Boylan
18275980f0 Check find's output in Jenkins' translation scripts.
* modules/jenkins/files/slave_scripts/propose_translation_update.sh:
Instead of evaluating find's output in test treat finds output as a
string and check if that string is null to determine if find found any
.po files. This is a more reliable way to check for the presence of .po
files as it does not result in confusing test.

* modules/jenkins/files/slave_scripts/upstream_translation_update.sh:
Instead of evaluating find's output in test treat finds output as a
string and check if that string is null to determine if find found any
.po files. This is a more reliable way to check for the presence of .po
files as it does not result in confusing test.

Change-Id: I53f7cc47e3762e4529eaf72f457852e650413b16
Reviewed-on: https://review.openstack.org/26346
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-04-08 20:19:16 +00:00
Clark Boylan
9c2072f525 Remove global python-subunit install.
The global python-subunit install that was used for subunit log
processing is interfering with the version desired for nova because nova
enabled sitepackages.

Remove the global install and use the local virtualenv install of
python-subunit to process subunit logs. This ensures compatibility with
testrepository and the subunit logs themselves.

Change-Id: Id634b7f96bcdb594242c0ae5734f77a9fdf5425c
Reviewed-on: https://review.openstack.org/26396
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
2013-04-08 18:45:49 +00:00
Clark Boylan
ac538fdb38 Fix comparison between Python Requirement objects.
pkg_resources.Requirement implements __eq__() but not __ne__(). There is
no implied relationship between __eq__() and __ne__() so we must negate
the result of == instead of using !=.

Change-Id: Ie83f87687f5666e6511265f2b9acdd42f38f1dd2
Reviewed-on: https://review.openstack.org/25974
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-04-03 16:55:53 +00:00
Clark Boylan
742e9cb9cc Compare requirements case insensitively.
Pip treats package names as case insensitive which has resulted in
different cases being used in the various project requirements files.
Use a case insensitive comparison when comparing a project against
openstack/requirements to avoid invalid comparison failures.

Change-Id: I59b3362ffb68d1c4662ac5d94919de47f0506218
Reviewed-on: https://review.openstack.org/25639
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-03-28 18:27:43 +00:00
James E. Blair
5c7c9b5396 Use new openstack-specific mirror.
Change-Id: I60dae939a1306c78d0da8ba6ae58fe4265ee97fb
Reviewed-on: https://review.openstack.org/23778
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-03-26 18:19:24 +00:00
James E. Blair
b7ceb913ad Make openstack mirror optional.
While we debug mirror creation.

Change-Id: Iae00bacded97d1d59ee3a48c3aa75d6b87708821
Reviewed-on: https://review.openstack.org/24712
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-03-18 21:32:13 +00:00
James E. Blair
8c6df9f19c Select a pypi mirror before running tox.
Change-Id: I62a86b6a0ad30ff8cb387ddcc05668128248c6d0
Reviewed-on: https://review.openstack.org/23513
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-03-18 20:29:07 +00:00
James E. Blair
e9bf54b03a Revert "Force python2.6 during python 2.6 jobs."
This reverts commit 80610a86a9c998aa876ae79f81523e0423363084

Change-Id: I3de4134669ff972c60797393783dd1d02763feb7
Reviewed-on: https://review.openstack.org/24493
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-03-15 00:18:19 +00:00
Monty Taylor
80610a86a9 Force python2.6 during python 2.6 jobs.
tox runs setup.py sdist outside of the venv because it's crazy. Until
we get that fixed, we need to let it know what python to use.

Change-Id: I90dd245dfcc3361fac0205394d207af2b0072638
Reviewed-on: https://review.openstack.org/24218
Reviewed-by: Anita Kuno <akuno@lavabit.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-03-14 23:03:39 +00:00
James E. Blair
f8cd914eb8 Add a requirements testing script.
Can be used to verify that updated requirements match those
specified in openstack/requirements.

Change-Id: Ieec680f0373786b75eaf947af15c60a78c763180
Reviewed-on: https://review.openstack.org/23843
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-03-07 22:29:46 +00:00
Robert Collins
cb38ca1361 Teach subunit2html to use v2 when it is available.
This changes the API of the result object to be more standard with
Python2.7+ APIs.

Change-Id: I1dd3da2280d687a591efd069ef43b1f2bc40e9f1
Reviewed-on: https://review.openstack.org/23670
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
2013-03-07 05:17:55 +00:00