gate-cinder-tox-compliance is broken because run-tox.sh script fails
with ".tox/compliance/bin/pip: No such file or directory". This is
because we're reusing pep8 venv (envdir tox directive), so
.tox/compliance directory never gets created.
To fix hat we can simply stop reusing that venv and let compliance tox
job create it's own venv. It can be done by removing envdir directive
from tox.ini.
Change-Id: Ib809872f5fb1581e1b098775c2ebdc4e724c70e2
Closes-Bug: 1596508
This is the start of an effort to both validate that drivers fully
implement the expected minimum requirements as well as to create a clear
place for driver developers to learn what needs to be implemented and get
documentation explaining what is expected for each method.
This also enables us to create tooling for documenting the available
drivers and their capabilities, to some degree. A follow up patch will
show some of what I'm thinking there, but it will make it possible to write
scripts for different needs.
This is somewhat a cleanup attempt to the ABC work that was started a
while back. This does not aim to replace that effort, but give a
mechanism for some of the things expected out of that effort that ended
up not being possible with how it evolved.
In most cases we do not really care if a driver is inherited from a
certain base class, just that it conforms to the given interface.
The interface/inheritance work really centers around two separate
things:
* Ensuring drivers conform to an expected interface
* Allowing code reuse and common implementation
This is really for the first item. Additional work is needed to complete
the ABC work we've done, but that really focuses on the second item, and
is out of scope for the intent of this patch.
Change-Id: I4168225126fe88c31712d94f0a130e9e7ede3446
The oslo sphinxconfiggen module was added to the oslo.config
2.3.0 release. This enables config file generation as part of
the sphinx doc generation.
The generated config file will pick up the current config
options from the code base. And as an added bonus, it will
now be published to the docs.openstack.org site for easy
reference or download.
This also puts us inline with what other projects like Nova
are doing for sample config files and is the recommended
method from the Oslo team.
Change-Id: I912a97eb2686d3dc56e50d8641d7bd930179bc18
Running the api-ref environment currently warns that 'rm' is not
installed in testenv. This patch removes the warning by whitelisting
the rm command.
Change-Id: I4072491713a38b11d9f55b26a8ba0a3cf49445f0
Closes-Bug: #1581778
This patch is for converting API Reference to RST and host it
in the Cinder tree.
This patch contains all the RST for cinder to bring over to their
repos to begin building API reference information from within.
This contains .inc files which have all the contents of the .rst files
but are grouped together for easier editing
This is the results of the RST conversion from WADL. It creates a
single index plus a bunch of included files which represent sections
of the API.
Cleaning task will be done once this part is merged.
As we have removed XML API support in Newton release, we need to
remove XML from API reference as well.
But we still have installations which are using XML.
So implementation plan is to first merge this patch in Newton release
keeping XML for now, then I will backport same to stable branches.
After that I will remove same from Newton release to match API
reference with actual API implementation.
Implements: bp api-reference-to-rst
Change-Id: I865ac922538bfa5bd45c24eb4bc49f5e966dc811
The most recent version of bandit no longer requires a configuration
file (bandit.yaml) to run the default test suite. Removing the
configuration file and updating the bandit arguments will run all of
the currently available bandit tests in the default test suite.
Change-Id: If144c4017ce9bb58c3e83b62a5e31325353200b5
Now that all cinder unit tests pass on python 3.4 [1], we can run py34
tests by default alongside py27 and pep8.
This commit also addresses the annoyance of py34 tox tests failing with
'db type could not be determined' if py27 tests were initially run in
the workspace.
[1] https://haypo.github.io/openstack_mitaka_python3.html
Change-Id: If60e5d0d3185e78f38fa2bfc7b6bb4840f09d840
Closes-bug: #1489059
This flips the tox.ini to always use constraints installation for
all targets. It drops the extra -constraints targets in the process.
This makes it so there is no developer change required to operate in our
new world order.
Change-Id: I2b56835edb2d75b7c871aae030e909578dbb958d
Depends-On: Iecd35d6bc3b1e9e0ed31b48a34e4149d34d55ac2
* Remove tests-py3.txt: all unit tests now pass on Python 3
* tox.ini: remove specific command for py34 test environment, it now
inherits testenv commands (same command than Python 2.7)
Partial-Implements: blueprint cinder-python3
Change-Id: I52c68a63740b3ae6097fad419b59a9c29422b525
Functional unit tests should not be a part of unit
tests job. This patch adds new tox environment to run
functional tests:
$ tox -e functional
It also removes logging from functional tests. Some of log calls were
replaced with asserts.
Related Implements: blueprint cinder-integrated-tests
Change-Id: I0ebfef2fe05f502cd5fb08fbc8af09008949e457
Even though testenvs pep8 and pep8-constraints included cinder/common
directory to be processed by flake8, global configuration in flake8
secion was excluding common glob preventing the directory from being
checked.
This patch removes glob from flake8 exclude section, unnecessary
cinder/common inclusion when running flake8, and fixes
cinder/common/config.py.
If we had a common directory in our code root and wanted to exclude it,
glob should be ./common instead of just common like we had now.
Change-Id: Iefc64c02335ed9296bda323fdcb5688125fb4e77
This allows discovery of dependencies that are
not listed in [test-]requirements.txt. Taken from
oslo.db.
Change-Id: I66000bb61e433f8d75ed15d600f2e87ef45140b8
Following the upgrade to hacking version 0.10.0 the H105 issues were
ignored. There was a note left in tox.ini to clean up the H105 issues
and to re-enable the check.
Updated tox.ini to re-enable the check of H105 and fixed corresponding
pep8 issues. This way Cinder aligns rows with Nova, Neutron, Glance etc
that have H105 enabled.
Closes-Bug: 1531104
Change-Id: I2da70cbeb838f16ec8479284dda9d7c62566dace
Currently, "tox -e pep8" builds two tox envs:
a "pep8" env, and then a "venv" env for genopts.
This is rather wasteful -- reuse the pep8 env
instead.
Change-Id: I8ba3d4e541f279e5bc62dba2b1caaac447fee7e3
2f9e416 Downstream Fix for Genconfig
collided with
fb17ed8 Add -constraints for CI jobs
while merging and this only got added to
pep8-constraints. It should be in "pep8" as well.
opts.py is updated here since it is out of sync,
due to this test not running.
Change-Id: I668df6be4301a052a4b47549c702a81b43abd8fc
This patch adds the opts.py file to the tree as a fix for deployers
that package cinder. The opts.py file is no longer being deleted right
away by generate_sample.sh after the cinder.sample.conf is being
generated.
This patch also introduces a pep8 check to make sure that the opts.py
file is up to date, so that it will catch when new opts get added to
Cinder without the opts.py being updated.
To support the ability to keep and check the opts file a number
of changes were needed in the check_uptodate.sh script as well as
the generate_sample.sh script:
- check_uptodate now takes --checkopts instead of --checkonly
When checkopts is used the opts.py file is generated using the
current code and the generated file is compared to the existing
file. The check fails if there are differences.
- generate_sample now has the --nosamplefile option.
When this option is used, only the opts.py file is generated.
The oslo-config-generator code is skipped so no sample file
is created.
- generate_sample also has some coding style consistency changes.
- Added the 'genopts' option to tox so users can generate
a fresh opts.py without a sample file when necessary.
Closes-Bug: 1501820
Co-Author: Jay Bryant <jsbryant@us.ibm.com>
Change-Id: I1f5494ebb19d5f4e8c651cbeef0acad07ad96829
Use the ObjectVersionChecker fixture from oslo.versionedobjects to check
when Cinder's registered objects have a change (new remotable method,
new field, etc) that require a version bump.
Fixes some registration problems along the way:
1. CinderObject doesn't need to be registered.
2. TestObject should be registered where it's used, not globally.
Removes deps=requirements.txt from tox.ini since it's not necessary and
prevents us from pip installing oslo.versionedobjects[fixtures].
Closes-Bug: #1514926
Change-Id: Id57c56a75ea11411e9e54104165bd44b577c1485
Other projects have updated to 1.8 or later for the minimum
tox version (nova, neutron), so we should probably do the
same.
Change-Id: Iddb9ab286519d555671ec39c0ce75c9351025730
This adds the constraints factor to the base section and duplicates
the sections required for CI with -constraints as tox does not
currently support factors in sections. Work towards enabling factor
support in sections is currently stalled, as such we will need to
duplicate sections adding -constraints to facilitate running sections
with the constrained install_command.
Implements Blueprint: Requirements-Management
Change-Id: I7afd52517750b82dcac94b8aae68dc0e94d00e54
This reuses the .tox/pep8 environment rather than
building a separate .tox/fast8 environment for
fast8.
This reduces duplicated install time and disk space.
Change-Id: I230465b28ba3d0f2c3a2ba38011ff9f7d472d719
"tox -e py34" now uses ostestr instead of testtools.run to run tests.
Tests are now executed in parallel.
Move the tests whitelist from tox.ini to a new file tests-py3.txt.
Fix NaElement.__str__() of netapp driver: return Unicode on Python 3,
not bytes.
Partial-Implements: blueprint cinder-python3
Change-Id: I4cda85fbe76ac853beaec3a9587726166daa063d
Modify assertRaisesAndMessageMatches() to tolerate subclasses. On
Python 3, a permission error now raises an exception PermissionError
which is a subclass of OSError, whereas the test expected exactly the
OSError type.
Partial-Implements: blueprint cinder-python3
Change-Id: Ia232c27de6a67eafad30b25301531e292fe349dc
Fix query_cluster_vols_for_ssc(): cast string (records) to int. On
Python 3, comparison between str and int now raise a TypeError.
tox.ini: add cinder.tests.unit.test_netapp_ssc to Python 3.4.
Partial-Implements: blueprint cinder-python3
Change-Id: I05127340029a5bb6c8444f51341be8008aab8dc8
* Add close() method to FakeHttplibSocket,
FakeDirectCmodeHTTPConnection, FakeDirect7modeHTTPConnection
* Replace StringIO() with BytesIO() for socket buffers
* makefile(): ignore optional arguments after mode
* Fix bytes versus Unicode: use bytes for HTTP body. Encode HTTP body
to UTF-8 and HTTP headers to Latin1.
* tox.ini: add cinder.tests.unit.test_netapp to Python 3.4
Partial-Implements: blueprint cinder-python3
Change-Id: I3dc8c36a84ffbb47be3e85eed34518c6adad48ec
* Rewrite ReadOnlyDict using collections.Mapping: collections.Mapping
has no method to modify the dictionary and is available on Python 2
and Python 3. Add also a __repr__() method to ease debug. Remove
ReadOnlyDict.update() method: a read-only dictionary must not be
modifiable.
* test_host_manager.py: sort dictionaries using a key function to
have a reliable order for the list of dictionaries. On Python 3,
the hash function is now randomized.
* tox.ini. add cinder/tests/unit/scheduler/ tests to Python 3.4
Blueprint cinder-python3
Change-Id: I34d98d6a75fef907251719f0d46b025d8e8b2b65
BlkioCgroup._set_limits(): sort devices before iterating on them to
have a reliable behaviour.
The devs variable is a dictionary. On Python 3, the hash function is
now randomized, so iterating on a dictionary gives items in a random
order. Use sorted() to iterate on the list of sorted devices instead.
tox.ini: add cinder.tests.unit.test_volume_throttling to Python 3.
Blueprint cinder-python3
Partial-Bug: #1348818
Change-Id: Icf7141f772397c7ac08f0f1e21ad74cb86a06351
* don't use hasttr() to check if a lazy SQLAlchemy is loaded or not: use
the obj_attr_is_set() method instead. On Python 3, hasattr() pass
through SQLAlchemy exceptions which is unexpected.
* Replace a/b with a//b to use integer division, not float division.
* Replace filter() with list-comprehension using an if.
* Replace file() with open() and open /dev/null in binary mode (not text
mode, so Unicode on Python 3).
* test_volume require "import cinder.volume.targets.tgt" on Python 3
* Use a key function to sort a list of dictionaries. Dictionaries are no
more comparable on Python 3.
* tox.ini: add the following tests to Python 3.4.
- cinder.tests.unit.keymgr.test_mock_key_mgr
- cinder.tests.unit.test_volume
Note: test_list_availability_zones_enabled_service() of test_volume
was broken, the test checked that the result of .sort() is None...
Blueprint cinder-python3
Change-Id: If1a26acc0138db9bda7fde1cb1f40093d9b3c494
* Replace func.func_name with func.__name__: the func_name attribute
was removed in Python 3, whereas the __name__ attribute exists on
Python 2 and Python 3
* Create INITIATOR_TARGET_MAP using list(set()) to get the same order
than the tested code. The exact order is not reliable, it depends
on the hash function which is now randomized by default on
Python 3. The hash function is also different between Python 2.7
and 3.4.
* tox.ini: add test_pure to Python 3.4
Blueprint cinder-python3
Change-Id: I78601278259f1d34ad6ac3458d2dd0a3aca9d77a
This replicates the run_tests.sh -8 behavior, running
flake8 only on changes made in the last commit and
working tree.
Change-Id: Iebd5746c78e840225f0860843c77f4c638877c26
As it stands, the opts.py file that is passed into
oslo-config-generator isn't being generated dynamically
and the old way of generating the cinder.conf.sample is
dependent on oslo-incubator which Cinder is trying to
move away from. oslo-config-generator works differently
than oslo-incubator so a number of changes had to be made
in order to make this switch.
This patch adds the config directory to Cinder and in it
are two files:
-generate_cinder_opts.py that will take the
results of a grep command to create the opts.py
file to be passed into oslo-config-generator.
-cinder.conf which is the new configuration for
oslo-config-generator. The file is inside the config
directory to be consistent with other projects.
Some changes were made to the generate_sample.sh file in
order to give the base directories and target directories
to the generate_cinder_opts.py program.
tox.ini was edited to remove the checkonly option because
all that needs to happen in check_uptodate.sh is a check to
ensure that the cinder.conf.sample is actually being
generated with no issues.
All options were removed from the check_uptodate.sh
because they were unnecessary given the new, more simple
way of generating the cinder.conf.sample.
setup.cfg was also edited in order to add information
oslo-config-generator needs to run.
Co-Authored By: Jay Bryant <jsbryant@us.ibm.com>
Co-Authored By: Jacob Gregor <jgregor@us.ibm.com>
Change-Id: I643dbe5675ae9280e204f691781e617266f570d5
Closes-Bug: 1473768
Closes-Bug: 1437904
Closes-Bug: 1381563
Port cinder.image_utils.check_qemu_img_version() to Python 3.
Add the following tests to Python 3.4 in tox.ini:
* cinder.tests.unit.test_image_utils
* cinder.tests.unit.test_migrations
* cinder.tests.unit.test_misc
* cinder.tests.unit.test_service
Partially implements: blueprint cinder-python3
Change-Id: I69feda94c90206adc8771fbd5ddc6aae5bb6b64b
