Systemd overrides have been added to the service unit files for all
beats and services. All of the playbooks have been updated to make them
look and feel uniform.
This also sets handlers within the playbooks so that we're improving the
idempotence.
Change-Id: I2dd3183dae4bfddc607cc74f9dfb7af115b80abc
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
To install curator facts are required, this change ensures we gather
them.
Change-Id: I510692095cdf8ecb5806a43c714b7bbbace47022
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The following options will reduce cluster pressure and generally
improve search performance.
Change-Id: I1619680db1fd595503f0845b182d6f6ce4c59f3c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The following change will ensure that the elastic-static is logging to
the journal and that systemd is able to report how well the elastic
slice is running.
Change-Id: I79a9074b5f14a41dec421d6691fd04c0e6be15b7
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Commit 875fa96f / change-id Ief0040f6 unintentionally tries to enlarge
the "machines00" LV when LXC is the default container technology which
fails due to the Debian automated installation having assigned all the
space within the associated "vmvg00" VG.
As the intention of the aforementioned commit was to apply when
systemd-nspawn was used, codify that explicitly in a `when:` condition
on the problematic Ansible task.
Change-Id: I56ec1290d71d0d09db447e347d7d55432d9b81c6
Signed-off-by: Corey Wright <corey.wright@rackspace.com>
Closes-Bug: #1781823
The curator retention policies will now query the storage nodes within
a given deployment and set a suitable index retention policy based on
the total amount of storage each index is assumed to produce every day.
To ensure we're minimizing the storage required and optimizing search
performance several actions are now being taken:
* Indexes will be shrunk after a quarter of their retention time.
* Indexes will be deleted should they exceed the retention time.
Change-Id: I8bf548620b5404d25deaadba8fda93452ef64fa0
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
With the more recent versions of ansible, we should now use
"is" instead of the "|" sign for the tests.
This should fix it.
Change-Id: I897b918785c34523688c450bec16661f0f6e496e
Currently if CONTAINER_TECH=nspawn is uses, Cinder and Swift
are unable to create volumes as space is fully allocated for
machines volume.
This shrinks machine00 mount to 8192 to make space for Cinder
and Swift volumes when using nspawn for the container tech.
Change-Id: Ief0040f638f0d3570557ac76fd5e0a8aee80df8d
Some of the variables used the templates are in the OSA group_vars,
or are composed using other variables from group_vars. These are
not accessible using the embedded ansible. In addition, a deployer
may have referenced group_vars in user_variables. Adding the whole
OSA group vars tree via a symlink covers all of these cases.
Change-Id: I7c842b0d41f24e7c192ab196eb2cfc133bb548a5
Detect if NFS is present by looking into the available mounts from
ansible facts.
Change-Id: I0e2d90d9e706ad4f6527484d96757b8578cb61bb
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
If kibana and elastic-logstash are the same node the deployment needs to
allow for storage nodes to co-exist with the front-end.
Change-Id: Icf9d26fefe015bd39f16387b4934e573783ed1ea
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The journalbeat process was restarting every playbook run which was not
required. This change moves the restart process to a handler which will
ensure we're not restarting the services when it's not required.
Change-Id: I4c0082d04d99c71c902ae39ee5ad9efc5074889f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Avoids using setuptools 34.x as using it can hit this bug
when packages are being upgraded:
https://github.com/pypa/setuptools/issues/951
Pinning to setuptools==33.1.1 also aligns to the
global-requirement-pins.txt in newton-eol
Change-Id: Ib33b828751c5a36d61448d148c5941beb6827c73
The journalbeat shipper will now have a "journald" tag attached to it
which will then be used by logstash to identify items and add tags to
them based on the systemd slice.
To accomodate this load the logstash config has been updated to better
handle blowback and scale according to the capability of the underlying
hardware.
The logstash grok files have been moved into the files directory which
was done to speed up the time it takes to ship files to a given host.
Originally the task used to copy these files was using the template
module however none of these files have anything templated within them
so there's no need to run them through the template engine.
Change-Id: I4f0e50ac491b595c0d276f6e7292d2c6e61baa22
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Systemd journals are linked from the containers down to the hosts
starting in Rocky. Prior to this Journalbeat must also be on all
containers.
Change-Id: Iaaef19e76c40ba9c1ad58164c20da46766abeee6
When journal_paths is empty the function sdJournal.NewJournal()
is used to open the journal for the host system only.
A single entry in journal_paths is opened with
sdJournal.NewJournalFrom[Dir|Files](), and multiple entries are opened
only with sdJournnalFromFiles().
Adding a single entry of /var/log/journal in the config file causes
all journal files under that directory to be opened, rather than only
that of the host system.
Change-Id: Ib758407edebff6786bf64fcf95328fb89912e3f6
As get-pip.py evolves based on pip 10, while we're still using
pip 9, changes in the way it can be used are causing problems.
For example, the ability to use --download is no longer there.
As such, let's pin to a known good version and leave it at that
until we no longer need to use this script. Version 3.2 maps to
pip 7.1.2 which fulfills our needs.
Change-Id: I49f6c9b238b42d4840c17af222e3bc82dfa6167f
Using tox for requirements management requires in-repo
requirements files for all our repositories. Rather than
do that, we make use of the tests repo to capture our
common requirements and use this to install them.
This reduces our review requirement rate and simplifies
maintenance for us for the tox config. It also makes it
usable with 'Depends-On', which is marvellous!
The tox requirements definitions for docs/releasenotes
builds are left in-place as those are standard entries
across the community. If that changes at some point, we
can re-assess those entries too.
Depends-On: https://review.openstack.org/579208
Change-Id: Ib03a2836de7271dcbccab7a0742ed98515de859b
In order to improve the readability and robustness of the mnaio feature
I have replaced the shell out to virsh tasks to use the virt module
where available. I have also created a vm-status play that will
hopefully help resolve SSH failures into the VMs. This play utilizes
the block/rescue/handler pattern to attempt to restart the VM once if
it fails the initial SSH check. Hopefully this will reduce the SSH
failures due to a suck VM. This adds a new variable called
vm_ssh_timeout which allows the deployer an easy place to override the
default timeout. The python-lxml package is needed for the virt module.
Change-Id: I027556b71a8c26d08a56b4ffa56b2eeaf1cbabe9
To resolve the warnings, we:
1. Use import_playbook instead of include for the conditional
playbook inclusions in site.yml. TIL that using include_playbook
does not work with conditionals.
2. Use include_tasks instead of include for the openstack-image-setup
task set inclusion.
3. Switch to using 'is' instead of '|' for tests.
Change-Id: I6d68bd4fecda122a77f7934842c3479a4c0792fd
After various attempts to make virt-manager work on a Mac
using brew and other methods, I found a way that's simpler
and much lighter on the client. This patch changes the
README to reflect this method.
Change-Id: I4c7594e7c0c371b4c0e417a78c8833262e479d22
The capabilities check is done on the host, so it only needs
to be executed once, not once for every VM on the host. This
patch eliminates the duplicated checking.
Change-Id: I2bc7ebbe699e5ace82c1bcbdfd8e917661054fef
Being able to save the images and re-use them on other hosts
is extremely useful to cut down deployment time. This patch
allows an MNAIO setup to be setup using a file-based backing
store, then have those saved and re-used on the same host or
on other hosts.
Change-Id: I491d04fb94352e37312891a9b9bd58093fdd00cf
When accessing the VM's on an MNAIO host and doing multiple
rebuilds, the SSH keys are constantly changing. This creates
a situation where keys constantly have to be deleted and
accepted which isn't very user-friendly.
Given that this tooling is used for test purposes, we can
disable the host key checks without being too concerned.
Change-Id: I3dd1221c4789b0ab8e895b22b05906456fc1fc8f
This patch implements the following style changes:
1. The 'environment' argument is placed in the same
location for all plays, making sure it's easier
to find.
2. The play tags are located in the same place, also
making sure they're easier to find.
3. The line breaks between tasks and plays are set
to be consistently 1 between tasks and 2 between
plays.
4. Given that there are no roles being used, the use
of pre/post tasks is converted to only using tasks.
Change-Id: I2e22c8360d65256b8e44ca1e310e0668a651196d
Instead of using the shell module, we use the file module. It
achieves the same result idempotently.
Change-Id: I84adb76ee5d5a9e7dd56c15f7cdf8e220b841b23
To improve the chances of success during builds, retries
are added to the package install tasks. Also, given that
we're using Ansible > 2.1.x, we forgo the with_items loop
for the package installs and just give the package module
the list so that it installs them all at once. Finally,
we ensure that the 'name' argument is used for all package
lists rather than the 'pkg' argument which is for apt only.
Change-Id: I5f27ea0b05c70f6c5396bd41dfe3cce54579ccb3
The documentation describes using each setup-*.yml playbook
individually. This helps to reduce the memory usage on the
deploy host and also makes the output more readable as you're
only having to read the output of a single meta-playbook when
looking at the task results. In this patch we make the setup
process use the same set of playbooks described in the docs.
Change-Id: I596ed599de2e4302a82f2401f8fdf57f97660060
Using the file-based backing store is currently undocumented. This
patch updates the README to include instructions for using it.
Change-Id: I228171b6619512874aece46f705ee1a922610cd5
The README is currently misleading and outdated. This corrects
a few things related to rebuilding the test environment.
Change-Id: I6c9b1698fb77ddfcb4c7ade6cd5a7a14a14c55e6
