46 Commits

Author SHA1 Message Date
Scott Solkhon
3300815529 Fix Hashi login method for approles
Moves Hashi Vault client login to use `auth.approle.login` as
current method is being deprecated in the next release.

```
DeprecationWarning: Call to deprecated function 'auth_approle'.
This method will be removed in version '0.12.0' Please use
the 'login' method on the 'hvac.api.auth_methods.approle'
class moving forward.
client.auth_approle(vault_role_id, vault_secret_id)
```

Change-Id: Ie5c1ebe99c8508336cc10944fdaa742ad7d1d85e
2022-09-02 16:42:15 +01:00
Michal Arbet
4838591c6c Add loadbalancer-config role and wrap haproxy-config role inside
This patch adds loadbalancer-config role
which is "wrapper" around haproxy-config
and proxysql-config role which will be added
in follow-up patches.

Change-Id: I64d41507317081e1860a94b9481a85c8d400797d
2022-08-09 12:15:49 +02:00
Michal Arbet
de973b81fa Add proxysql support for database
Kolla environment currently uses haproxy
to fullfill HA in mariadb. This patch
is switching haproxy to proxysql if enabled.

This patch is also replacing mariadb's user
'haproxy' with user 'monitor'. This replacement
has two reasons:
  - Use better name to "monitor" galera claster
    as there are two services using this user
    (HAProxy, ProxySQL)
  - Set password for monitor user as it's
    always better to use password then not use.
    Previous haproxy user didn't use password
    as it was historically not possible with
    haproxy and mariadb-clustercheck wasn't
    implemented.

Depends-On: https://review.opendev.org/c/openstack/kolla/+/769385
Depends-On: https://review.opendev.org/c/openstack/kolla/+/765781
Depends-On: https://review.opendev.org/c/openstack/kolla/+/850656

Change-Id: I0edae33d982c2e3f3b5f34b3d5ad07a431162844
2022-07-29 15:05:21 +02:00
k-s-dean
8553e52acd adds firewalld configuration based on enabled services
This change introduces automated configuration of firewalld and adds
a new filter for extracting services from the project_services dict.
the filter selects any enabled services and their haproxy element
and returns them so they can be iterated over.
This commit also enables automated configuration of firewalld from enabled
openstack services and adds them to the defined zone and reloads the
system firewall.

Change-Id: Iea3680142711873984efff2b701347b6a56dd355
2022-07-27 12:28:40 +01:00
Marcin Juszkiewicz
333c74feb4 genpwd: handle lack of password file nicer
From:

(kolla) 13:11 (s) marcin@puchatek:kolla-ansible$ kolla-genpwd
Traceback (most recent call last):
  File "/home/marcin/.virtualenvs/kolla/bin/kolla-genpwd", line 8, in <module>
    sys.exit(main())
  File "/home/marcin/.virtualenvs/kolla/lib/python3.10/site-packages/kolla_ansible/cmd/genpwd.py", line 135, in main
    genpwd(passwords_file, length, uuid_keys, ssh_keys, blank_keys,
  File "/home/marcin/.virtualenvs/kolla/lib/python3.10/site-packages/kolla_ansible/cmd/genpwd.py", line 59, in genpwd
    with open(passwords_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/kolla/passwords.yml'

To:

(kolla) 13:17 (s) marcin@puchatek:kolla-ansible$ kolla-genpwd
ERROR: Passwords file "/etc/kolla/passwords.yml" is missing

Change-Id: I18a9559daeb3d124a03dcb735ebb01a2cf24f617
2022-05-24 13:18:20 +02:00
Zuul
087d0e1d60 Merge "neutron: add ssh key" 2022-03-30 13:38:32 +00:00
Michal Nasiadka
7fcf3ca30b neutron: add ssh key
This key can be used by users in networking-generic-switch
scenario instead of adding cleartext password in ml2_conf.ini.

Change-Id: I10003e6526a55a97f22678ab81c411e4645c5157
2022-03-30 07:28:37 +00:00
Zuul
60f8b7410d Merge "re-add rabbitmq config for clustering interface" 2022-03-25 09:52:10 +00:00
Sven Kieske
1599252483
re-add rabbitmq config for clustering interface
this adds back the ability to configure
the rabbitmq/erlang kernel network interface
which was removed in https://review.opendev.org/#/c/584427/
seemingly by accident.

Closes-Bug: 1900160

Change-Id: I6f00396495853e117429c17fadfafe809e322a31
2022-03-24 17:52:17 +01:00
Mark Goddard
fc2292b230 Use jinja2.pass_context instead of contextfilter
The contextfilter decorator was deprecated in jinja2 3.0.0, and has been
dropped in 3.1.0. This results in the following warning, and failed
attempts to use filters:

    [WARNING]: Skipping plugin (filters.py) as it seems to be invalid:
    module 'jinja2' has no attribute 'contextfilter'

This change switches to use the pass_context decorator. The minimum
version of Jinja2 is raised to 3 to ensure pass_context is present.

Change-Id: I649dd6211d3ae72b9539bc44652ef8cf5d579777
2022-03-24 16:29:26 +00:00
wu.shiming
b218b29c38 Replace deprecated assertRaisesRegexp
The assertRaisesRegexp method has been deprecated since it was renamed
to assertRaisesRegex in Python 3.2.

https://docs.python.org/3/library/unittest.html#deprecated-aliases

Change-Id: I38ed4bebee3617267463d13d8f12bc083ab74ac2
2021-11-09 09:02:43 +08:00
Zuul
2ecf0a8783 Merge "Support storing passwords in Hashicorp Vault" 2021-07-02 20:28:44 +00:00
Scott Solkhon
6bf74aa20d Support storing passwords in Hashicorp Vault
This commit adds two new cli commands to allow an operator
to read and write passwords into a configured Hashicorp Vault
KV.

Change-Id: Icf0eaf7544fcbdf7b83f697cc711446f47118a4d
2021-06-30 15:16:12 +01:00
Mark Goddard
ade5bfa302 Use ansible_facts to reference facts
By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements
2021-06-23 10:38:06 +01:00
Kendall Nelson
25b9de91a2 Remove Retired Karbor Support
As announced on the openstack-discuss ML[1], Karbor is retiring
this cycle (Wallaby).

Needed-By: https://review.opendev.org/c/openstack/karbor/+/767032

[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018643.html

Change-Id: I222cf302e507f6a9de0347c79ec536aa7be22bb6
2020-12-22 09:50:49 +00:00
wu.chunyang
4a58f4238c Implement automatic deploy of octavia
this patchset has implemented:
  - network (lb-mgmt-net)
  - security groups and rules (used by amphora and health manager)
  - amphora flavor (used by amphora)
  - nova keypair (used by amphora at the time of debugging)

Add a octavia_amp_listen_port variable which used by amphora
Add amp_image_owner_id in octavia.conf

Implements: blueprint implement-automatic-deploy-of-octavia
Co-Authored-By: zhangchun <zhangchun@yovole.com>

Depends-On: https://review.opendev.org/652030

Change-Id: I67009d046925cfc02c1e0073c80085c1471975f6
2020-10-02 14:05:00 +02:00
Zuul
fbef9b36d6 Merge "Performance: use a single config file for fluentd" 2020-09-17 11:03:26 +00:00
Zuul
00ed275c44 Merge "Fix kolla_address in IPv6 fully-routed topo case" 2020-08-01 04:14:34 +00:00
Mark Goddard
56a07702bc Performance: use a single config file for fluentd
Currently we generate multiple fluentd configuration files for inputs,
filters, formatters and outputs.
These are then included from the main td-agent.conf configuration file.
With a large number of hosts, this can take a long time to template.

Benchmarking of templating is available at [1].

This change switches to a single fluentd configuration file, with the
include done locally. For the default template files included with Kolla
Ansible we use Jinja includes, but this does not work with templates in
a different directory. We therefore use the Ansible template lookup
plugin, which has a slightly higher overhead than a jinja include, but
far lower than generating multiple templates. This should drastically
improve the performance of this task.

[1] https://github.com/stackhpc/ansible-scaling/blob/master/doc/template.md

Partially-Implements: blueprint performance-improvements

Change-Id: Ia8623be0aa861fea3e54d2c9e1c971dfd8e3afa9
2020-07-28 12:14:48 +01:00
Mark Goddard
620b808cf3 Improve error reporting in password utilities
The kolla-genpwd and kolla-mergepwd commands can be used to manipulate
the kolla passwords.yml file. The format is a YAML encoded dict of
password variable names to their values. If the format is not a dict,
the error messages are unhelpful.  In particular, this can happen if the
file is encrypted e.g. via Ansible Vault.

For kolla-genpwd:

    AttributeError: 'NoneType' object has no attribute 'items'

For kolla-mergepwd:

    AttributeError: 'NoneType' object has no attribute 'update'

This change adds a more friendly message.

Change-Id: I27f0835b904e05006ae401adf383090322e1b891
Closes-Bug: #1880220
2020-05-22 19:20:47 +01:00
Radosław Piliszek
0674236eed Fix kolla_address in IPv6 fully-routed topo case
This includes some lightweight refactoring to avoid code
duplication.

This patch is made to be backportable to Train.
We now include Ansible in testing since Ussuri so the comments
about the bool filter are wrong.

Change-Id: Ia2e0f7f24988763bacfeafefb7977021f5949f4e
Closes-bug: #1848941
2020-05-19 19:49:14 +02:00
gugug
22f7aecbd9 Enable W503 for flake8 check
W503 and W504 are incompatible and we need to choose one of them.
Existing codes follows W503, so we disable W504.

Change-Id: Ic745e956dd332eb0fa49b93c1e6acb12f8a7f26c
2020-05-17 18:21:39 +08:00
Mark Goddard
8cc58e3669 Fix service_mapped_to_host filter for common services
In Ibecac60d1417269bbe25a280996ca9de6e6d018f, the services in the common
role were marked as being mapped to the 'all' group, since the
'service_mapped_to_host' filter expects every service definition to have
either a 'group' or 'host_in_groups' field. While this allows the filter
to pass the common services without error, it will not actually show
them as being mapped to any hosts. This is because the filter uses the
'group_names' variable, which contains all of the groups that a host
belongs to, except the default 'all' group.

This change fixes the issue by returning True from
service_mapped_to_host when the service's group is 'all'.

Change-Id: I39c8416f5d30a535c1743f9c43434b7d2a382196
Related-Bug: #1868596
2020-04-14 14:15:54 +01:00
Andreas Jaeger
454489762c Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found by updated hacking version.

Remove hacking and friends from lower-constraints, they are not needed
during installation.

Change-Id: I7ef5ac8a89e94f5da97780198619b6facc86ecfe
2020-04-04 17:00:17 +02:00
Radosław Piliszek
5648ad6e9d Use unittest.mock instead of PyPI mock
Now that py2 is gone, oslotest dropped dependency on mock and will
soon affect Ussuri CI [1], let's use unittest.mock built in py3.

This also fixes py38 jobs and proactively prevents py36 and py37
failing due to [1]. This is because we never included mock in
test-requirements (but in lower-constraints where it does not
really belong at all) and instead relied on oslotest to bring
it in.

[1] https://review.opendev.org/716322

Change-Id: I30e82e2d87418272a71c7ee089a8acdaf8872158
2020-04-02 18:21:01 +02:00
Mark Goddard
35966c9186 Fix service_mapped_to_host filter
The service_mapped_to_host filter is used to check if a service is
mapped to a host, based on the group for the service or its
host_in_groups attribute if one exists. We check if the service's group
is in the 'groups' list. However, to get the list of groups to which a
host belongs, we should use the 'group_names' list.

This filter is currently only used in neutron IPv6 module loading, so
the effects are minimal.

Change-Id: I37409ca8d273b0426df0a648db222dc5432e738a
Closes-Bug: #1868285
2020-03-20 17:03:14 +00:00
Zuul
cd3c51197e Merge "Remove kolla-ceph" 2020-02-13 10:09:24 +00:00
Michal Nasiadka
4e6fe7a6da Remove kolla-ceph
Kolla-Ansible Ceph deployment mechanism has been deprecated in Train [1].

This change removes the Ansible code and associated CI jobs.

[1]: https://review.opendev.org/669214

Change-Id: Ie2167f02ad2f525d3b0f553e2c047516acf55bc2
2020-02-11 11:42:06 +01:00
Radosław Piliszek
ed225a36d8 Add --clean to kolla-mergepwd
to clean old keys on merge.

Change-Id: Ifcc99e7c737707eea9e951db066dc94fd85bd9f7
2020-02-08 16:28:33 +01:00
Zuul
44d998e506 Merge "Add unit tests for Nova Cells filters" 2019-10-21 14:48:32 +00:00
Zuul
486aa8c212 Merge "Fix Python3 compatibility for kolla-genpwd" 2019-10-21 10:39:19 +00:00
Doug Szumski
8e068099de Add unit tests for Nova Cells filters
This moves the Nova Cells filters alongside the service filters
for ease of testing.

Partially Implements: blueprint support-nova-cells
Change-Id: I32d35c065812c6b46c64bacdf283a0bdad0f8a0f
2019-10-18 13:13:32 +00:00
Radosław Piliszek
bc053c09c1 Implement IPv6 support in the control plane
Introduce kolla_address filter.
Introduce put_address_in_context filter.

Add AF config to vars.

Address contexts:
- raw (default): <ADDR>
- memcache: inet6:[<ADDR>]
- url: [<ADDR>]

Other changes:

globals.yml - mention just IP in comment

prechecks/port_checks (api_intf) - kolla_address handles validation

3x interface conditional (swift configs: replication/storage)

2x interface variable definition with hostname
(haproxy listens; api intf)

1x interface variable definition with hostname with bifrost exclusion
(baremetal pre-install /etc/hosts; api intf)

neutron's ml2 'overlay_ip_version' set to 6 for IPv6 on tunnel network

basic multinode source CI job for IPv6

prechecks for rabbitmq and qdrouterd use proper NSS database now

MariaDB Galera Cluster WSREP SST mariabackup workaround
(socat and IPv6)

Ceph naming workaround in CI
TODO: probably needs documenting

RabbitMQ IPv6-only proto_dist

Ceph ms switch to IPv6 mode

Remove neutron-server ml2_type_vxlan/vxlan_group setting
as it is not used (let's avoid any confusion)
and could break setups without proper multicast routing
if it started working (also IPv4-only)

haproxy upgrade checks for slaves based on ipv6 addresses

TODO:

ovs-dpdk grabs ipv4 network address (w/ prefix len / submask)
not supported, invalid by default because neutron_external has no address
No idea whether ovs-dpdk works at all atm.

ml2 for xenapi
Xen is not supported too well.
This would require working with XenAPI facts.

rp_filter setting
This would require meddling with ip6tables (there is no sysctl param).
By default nothing is dropped.
Unlikely we really need it.

ironic dnsmasq is configured IPv4-only
dnsmasq needs DHCPv6 options and testing in vivo.

KNOWN ISSUES (beyond us):

One cannot use IPv6 address to reference the image for docker like we
currently do, see: https://github.com/moby/moby/issues/39033
(docker_registry; docker API 400 - invalid reference format)
workaround: use hostname/FQDN

RabbitMQ may fail to bind to IPv6 if hostname resolves also to IPv4.
This is due to old RabbitMQ versions available in images.
IPv4 is preferred by default and may fail in the IPv6-only scenario.
This should be no problem in real life as IPv6-only is indeed IPv6-only.
Also, when new RabbitMQ (3.7.16/3.8+) makes it into images, this will
no longer be relevant as we supply all the necessary config.
See: https://github.com/rabbitmq/rabbitmq-server/pull/1982

For reliable runs, at least Ansible 2.8 is required (2.8.5 confirmed
to work well). Older Ansible versions are known to miss IPv6 addresses
in interface facts. This may affect redeploys, reconfigures and
upgrades which run after VIP address is assigned.
See: https://github.com/ansible/ansible/issues/63227

Bifrost Train does not support IPv6 deployments.
See: https://storyboard.openstack.org/#!/story/2006689

Change-Id: Ia34e6916ea4f99e9522cd2ddde03a0a4776f7e2c
Implements: blueprint ipv6-control-plane
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-10-16 10:24:35 +02:00
Ionut Balutoiu
e3fccdfa65 Fix Python3 compatibility for kolla-genpwd
The method `Fernet.generate_key()` generates a binary string in Python 3:
```
>>> Fernet.generate_key()
b'qSMZlOK23pZUw_Uyy-ZRPUfPskMXKGCGmhG6AHCFiV8='
```

Unless properly written as a string to the Kolla `passwords.yml` file,
the Fernet key will end up in the final Barbican config like this:
```
[simple_crypto_plugin]
kek = b'qSMZlOK23pZUw_Uyy-ZRPUfPskMXKGCGmhG6AHCFiV8='
```

Due to the fact that the key is incorrectly written to the barbican
config file (it should be written as a string), every barbican secret
store fails with:

```
barbican.api.controllers   File "/var/lib/kolla/venv/lib/python3.6/site-packages/barbican/plugin/store_crypto.py", line 83, in store_secret
barbican.api.controllers     encrypting_plugin, context.project_model)
barbican.api.controllers   File "/var/lib/kolla/venv/lib/python3.6/site-packages/barbican/plugin/store_crypto.py", line 290, in _find_or_create_kek_objects
barbican.api.controllers     kek_meta_dto = plugin_inst.bind_kek_metadata(kek_meta_dto)
barbican.api.controllers   File "/var/lib/kolla/venv/lib/python3.6/site-packages/barbican/plugin/crypto/simple_crypto.py", line 104, in bind_kek_metadata
barbican.api.controllers     encryptor = fernet.Fernet(self.master_kek)
barbican.api.controllers   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cryptography/fernet.py", line 38, in __init__
barbican.api.controllers     "Fernet key must be 32 url-safe base64-encoded bytes."
barbican.api.controllers ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
```

This commit fixes the issue described above by properly writing
the Fernet key as a string to the Kolla `passwords.yml` file.

Closes-Bug: #1848191
Change-Id: I27fc0159c889bc2e1576fdd69b7d02a320b620f8
2019-10-15 11:27:07 +00:00
Mark Goddard
af2e7fd73e Add custom filters for checking services
These filters can be used to capture a lot of the logic that we
currently have in 'when' statements, about which services are enabled
for a particular host.

In order to use these filters, it is necessary to install the
kolla_ansible python module, and not just the dependencies listed in
requirements.txt. The CI test and quickstart install from source
documentation has been updated accordingly.

Ansible is not currently in OpenStack global requirements, so for unit
tests we avoid a direct dependency on Ansible and provide fakes where
necessary.

Change-Id: Ib91cac3c28e2b5a834c9746b1d2236a309529556
2019-09-16 12:48:52 +00:00
Maciej Kucia
89e91b69bd cmd: Extract methods to allow import from external
When methods for passwords generation and merge are
extracted then external apps and scripts can use
those methods without resolving to subprocess execution
or injecting sys.argv.

Change-Id: I99aff7852180534129fa36859075306eea776ba9
Signed-off-by: Maciej Kucia <maciej@kucia.net>
2019-03-10 21:02:38 +01:00
Eduardo Gonzalez
b80a63f33f Use fernet for barbican crypto key
Sha password is not always valid for barbican cripto key.
Use a fernet key so it always gets valid.

Not need release note for upgrade, users with a working
barbican not regenerate passwords, only new passwords will
get new type.

Change-Id: Ic8c4ca63219295d697062cff9cbf30fadbe49bf3
2018-07-26 22:01:30 +02:00
Borne Mace
f1768ef7ab Updated genpwd to work with python35
Due to the changes in hmac.new and how binary strings
are dumped in yaml.safe_dump some changes were needed to
make sure that we dumped only strings, not binary strings.

Change-Id: Ic2fbcf2347023c1e9e666203dfe40dbeaf24ce5f
2018-05-22 16:33:40 -07:00
Christian Berendt
bc0f52cdc7 Add missing dot to help string in genpwd.py script
Change-Id: I87df49939f600cfa1041193808ce6bdcf4620ffc
2017-09-14 00:16:19 +02:00
Jenkins
1529d4e54e Merge "Use cryptography instead of pycrypto" 2017-06-19 10:44:53 +00:00
Eduardo Gonzalez
ab4b1ff785 Support OSprofile usage
OSprofile allows user/devs trace OpenStack requests.

Implements: blueprint enable-osprofiler
Co-Authored-By: Bertrand Lallau <bertrand.lallau@gmail.com>
Change-Id: I82ea85d726011ef6cbf99380f395452d6d7f8053
2017-06-02 22:41:33 +02:00
Rui Yuan Dou
43d42d07df Use cryptography instead of pycrypto
pycrypto is no longer maintained [1]. This patch rewrites functions
using pycrypto and replaces them with the cryptography equivalent

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-March/113568.html

Change-Id: I375b5876ec2f4c4f32b9f6b3f41d209a59a0f615
2017-04-24 17:30:13 +08:00
zhuzeyu
a1e2901c30 Use yaml.safe_dump() instead of yaml.dump()
Remove Python specific types from YAML output
Produce only basic YAML tags

Change-Id: Ib6a4c18663897efb7243ed1ff84df1c9f2abf8bf
2017-03-30 16:54:05 +08:00
zhubingbing
6d0e31f232
Fix can't find /usr/lib/libCryptoki2_64.so in barbican
Link https://docs.openstack.org/project-install-guide/key-manager/newton/barbican-backend.html#simple-crypto-plugin

Change-Id: I351738c2a98090c56ac69e477fbe5ddec4cc5b26
Closes-Bug: #1672001
2017-03-22 20:43:14 +08:00
Jeffrey Zhang
d06efcecc5 Fix booting from volume failure
Booting from volume require cinder's ceph client secret now. Move cinder
before nova in site.yml, because nova depends on cinder ceph client key
now.

Change-Id: I01c9ed80843d98305b8963894c4917c21a35d3ac
Closes-Bug: #1670676
2017-03-08 21:16:06 +08:00
Jeffrey Zhang
177fbea79a Rename kolla namespace to kolla_ansible
* Rename kolla namespace to kolla_ansible
* remove oslo.config.opts entry points which is uesless
* delete useless tools/version-check.py script

Change-Id: I005dd7223ff23afbb2ce8cbfd0ebec0969102798
2017-02-15 16:34:51 +08:00