In some situations it may be helpful to populate the fact cache on
demand. The 'kolla-ansible gather-facts' command may be used to do this.
One specific case where this may be helpful is when running kolla-ansible
with a --limit argument, since in that case hosts that match the limit
will gather facts for hosts that fall outside the limit. In the extreme
case of a limit that matches only one host, it will serially gather
facts for all other hosts. To avoid this issue, run 'kolla-ansible
gather-facts' without a limit to populate the fact cache in parallel
before running the required command with a limit.
Change-Id: I79db9bca23aa1bd45bafa7e7500a90de5a684593
To follow best security practices and help fellow operators.
More details inline and in the linked bug report.
Closes-Bug: #1940547
Change-Id: Ide9e9009a6e272f20a43319f27d257efdf315f68
Basically, there are three main installation scenario:
Scenario 1:
Ironic installation together with other openstack services
including keystone. In this case variable enable_keystone
is set to true and keystone service will be installed
together with ironic installation. It is possible realise this
scenario, no fix needed
Scenario 2:
Ironic installation with connection to already installed
keystone. In this scenario we have to set enable_keystone
to “No” to prevent from new keystone service installation
during the ironic installation process. But in other hand,
we need to have correct sections in ironic.conf to provide
all information needed to connect to existing keystone.
But all sections for keystone are added to ironic.conf only
if enable_keystone var is set to “Yes”. It isn’t possible
to realise this scenario. Proposed fix provide support for
this scenario, where multiple regions share the same
keystone service.
Scenario 3:
No keystone integration. Ironic don't connect to Keystone.
It is possible realise this scenario, no fix needed
Proposed solution also keep the default behaviour: if no
enable_keystone_integration is manually defined by default
it takes value of enable_keystone variable and all behaviour
is the same. But if we don't want to install keystone and
want to connect to existing one at the same time, it will be
possible to set enable_keystone var to “No”
(preventing keystone from installation) and at the same
time set ironic_enable_keystone_integration to Yes to allow
needed section appear in ironic.conf through templating.
Change-Id: I0c7e9a28876a1d4278fb2ed8555c2b08472864b9
Multiple inventories can now be passed to `kolla-ansible`. This can be
useful to construct a common inventory that is shared between multiple
environments.
Change-Id: I2ac5d7851b310bea2ba362b353f18c592a0a6a2e
As mentioned in the Iced014acee7e590c10848e73feca166f48b622dc
commit message, in Ussuri+ we can use ``+sbwtdcpu none
+sbwtdio none`` as well. This is due to relying on RMQ-provided
erlang in version 23.x.
This change adds the extra arguments by default.
It should be backported down to Ussuri before we do a release with
Iced014acee7e590c10848e73feca166f48b622dc.
Change-Id: I32e247a6cb34d7f6763b544f247fd408dce2b3a2
In Ussuri, nova stopped using separate Ceph keys for the volumes and vms
pools by default. Instead, we set ceph_nova_keyring to the value of
ceph_cinder_keyring by default, which is ceph.client.cinder.keyring.
This is in line with the Ceph OpenStack integration guide [1]. However,
the user used by nova to access the vms pool (ceph_nova_user) defaults
to nova, meaning that nova will still try to use a
ceph.client.nova.keyring, which probably does not exist. We did not see
this issue in CI, because we set ceph_nova_user to cinder.
This change fixes the issue by setting ceph_nova_user to the value of
ceph_cinder_user by default, which is cinder.
Closes-Bug: #1934145
Related-Bug: #1928690
[1] https://docs.ceph.com/en/latest/rbd/rbd-openstack/
Change-Id: I6aa8db2214e07906f1f3e035411fc80ba911a274
Nova always tries to create the rabbitmq user regardless of
whether RabbitMQ is enabled or not.
This ps also adds an external rabbitmq doc.
Change-Id: Iec517226e4c82ea351889b55689a3efceaadcc76
In the Xena release, Ironic removed the iSCSI driver [1]. The
recommended driver is direct, which uses HTTP to transfer the disk
image. This requires an HTTP server, and the simplest option is to use
the one currently deployed when enable_ironic_ipxe is set to true. For
this reason, this patch always enables the HTTP server running on the
conductor.
iPXE is still enabled separately, since it cannot currently be used at
the same time as PXE.
[1] https://review.opendev.org/c/openstack/ironic/+/789382
Change-Id: I30c2ad2bf2957ac544942aefae8898cdc8a61ec6
The variable octavia_amphora_flavor should be octavia_amp_flavor.
The variable for customising network and subnet was only mentioned in
the example.
Change-Id: I3ba5a7ccc2c810fea12bc48584c064738e5aa35e
Ansible facts can have a large impact on the performance of the Ansible
control host. This patch introduces some control over which facts are
gathered (kolla_ansible_setup_gather_subset) and which facts are stored
(kolla_ansible_setup_filter). By default we do not change the default
values of these arguments to the setup module. The flexibility of these
arguments is limited, but they do provide enough for a large performance
improvement in a typical moderate to large OpenStack cloud.
In particular, the large complex dict fact for each interface has a
large effect, and on an OpenStack controller or hypervisor there may be
many virtual interfaces. We can use the kolla_ansible_setup_filter
variable to help:
kolla_ansible_setup_filter: 'ansible_[!qt]*'
This causes Ansible to collect but not store facts matching that
pattern, which includes the virtual interface facts. Currently we are
not referencing other facts matching the pattern within Kolla Ansible.
Note that including the 'ansible_' prefix causes meta facts module_setup
and gather_subset to be filtered, but this seems to be the only way to
get a good match on the interface facts. To work around this, we use
ansible_facts rather than module_setup to detect whether facts exist in
the cache.
The exact improvement will vary, but has been reported to be as large as
18x on systems with many virtual interfaces.
For reference, here are some other tunings tried:
* Increased the number of forks (great speedup depending of the size of
the deployment)
* Use `strategy = mitogen_linear` (cut processing time in half)
* Ansible caching (little speed up)
* SSH tunning (little speed up)
Co-Authored-By: Mark Goddard <mark@stackhpc.com>
Closes-Bug: #1921538
Change-Id: Iae8ca4aae945892f1dc65e1b10381d2e26e88805
Adds a new variable, 'disable_firewall', which defaults to true. If set
to false, then the host firewall will not be disabled during
kolla-ansible bootstrap-servers.
Change-Id: Ie5131013012f89c8c3b91ca359ad17d9cb77efc8
This commit adds two new cli commands to allow an operator
to read and write passwords into a configured Hashicorp Vault
KV.
Change-Id: Icf0eaf7544fcbdf7b83f697cc711446f47118a4d
By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.
This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.
This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.
[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars
Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements
On machines with many cores, we were seeing excessive CPU load on systems
that were not very busy. With the following Erlang VM argument we saw
RabbitMQ CPU usage drop from about 150% to around 20%, on a system with
40 hyperthreads.
+S 2:2
By default RabbitMQ starts N schedulers where N is the number of CPU
cores, including hyper-threaded cores. This is fine when you assume all
your CPUs are dedicated to RabbitMQ. Its not a good idea in a typical
Kolla Ansible setup. Here we go for two scheduler threads.
More details can be found here:
https://www.rabbitmq.com/runtime.html#scheduling
and here:
https://erlang.org/doc/man/erl.html#emulator-flags
+sbwt none
This stops busy waiting of the scheduler, for more details see:
https://www.rabbitmq.com/runtime.html#busy-waiting
Newer versions of rabbit may need additional flags:
"+sbwt none +sbwtdcpu none +sbwtdio none"
But this patch should be back portable to older versions of RabbitMQ
used in Train and Stein.
Note that information on this tuning was found by looking at data from:
rabbitmq-diagnostics runtime_thread_stats
More details on that can be found here:
https://www.rabbitmq.com/runtime.html#thread-stats
Related-Bug: #1846467
Change-Id: Iced014acee7e590c10848e73feca166f48b622dc
In the Xena cycle it was decided to remove the Monasca
Grafana fork due to lack of maintenance. This commit removes
the service and provides a limited workaround using the
Monasca Grafana datasource with vanilla Grafana.
Depends-On: I9db7ec2df050fa20317d84f6cea40d1f5fd42e60
Change-Id: I4917ece1951084f6665722ba9a91d47764d3709a
