This change adds the scaffolding required to get multi-distro support
running in the roles. The change breaks up our playbooks converting all
of the tasks into various roles with internal dependencies. While this
will improve execution time, the change is being done to reduce boiler
plate and to allow us to build on the pattern used in OSA to provide
multi-distro capabilities.
A side effect of this change is a major improvement in idempotency. The
playbooks should now be 100% idempotent.
All of the templates have been left in the main playbook directory. This
was done to help ease the transition. In a future PR the template
structure will be moved into the roles where it needs to be.
The main variable files has been left intact. This file will be carved
up into role defaults in a future PR.
Change-Id: I938a10564128ce4078fa12edcf614dcdbd684b25
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Systemd overrides have been added to the service unit files for all
beats and services. All of the playbooks have been updated to make them
look and feel uniform.
This also sets handlers within the playbooks so that we're improving the
idempotence.
Change-Id: I2dd3183dae4bfddc607cc74f9dfb7af115b80abc
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The templates which generate the configuration for the various
components require access to the facts defining the cluster layout.
Running the setup*.yml playbooks with --tags config would not include
common_task_data_node_hosts.yml and templating would fail due to
undefined vars.
This change applys an 'always' tag to all uses of the include.
Change-Id: I6c55a60aef54f9bb26eb9a2af08315265daa82d9
This change breaks out the basic install and setup playbooks into
different playbooks. This is done to ensure an operator can easily rerun
any part of the playbook as they upgrade, change, or modify a setup over
the lifetime of the deployment.
Change-Id: I66c53fcb21880c950ea3fee202e7d2224dfdff3a
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The tasks with external deps should all have a retry so to cater for
cases of intermittent service interruptions. This change adds retires
and tags to of the tasks with external deps.
Change-Id: Icd0a8143347bcbffc2a1e0a02da92091b59ed483
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Elasticsearch can be used as a smart load balancer for all traffic
which will remove the requirement for a VIP and move the cluster to a
mesh topology. All of the Kibana nodes will now run elasticsearch as
cordonator.
* Kibana will now connect to elasticsearch on localhost.
* All of the beats have been setup to use use the new mesh topology.
* jvm memory management has been updated to reflect the additional
services.
More on node assigments can be found here:
* https://www.elastic.co/guide/en/elasticsearch/reference/6.2/modules-node.html#modules-node
* The readme has been updated to reflect these changes.
Change-Id: I769e0251072f5dbde56fcce7753236d37d5c3b19
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
At present we're collecting too much info by default. We're seeing
+500GB on a <50 node environment in just two weeks. While we dont expect
the data set to grow much larger given the use of curator, this change
lowers the default collection intervals of the various beats and updates
the retention / detection policies so we're not storing too much
information.
To correct a unicode problem with py2 the host index loops have been
updated.
Curator has also been updated to run everyday.
Change-Id: Ic202eb19806d1b805fa314d3d8bde05b286740e0
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
When reconfiguring beats the tag makes it's a quick playbook run.
Change-Id: I22c9f7be39d6cf733800ee271bd299c457e3ee55
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
* Added options for the rollback plan so that if a rollback is executed
all beat packages will be removed.
* additional updates to streamline elk and fix container bindmounts,
the use of group information for metric and heartbeat information.
* Readme information has been fixed
Change-Id: Icd070259db5b19d289d10033b1f055125f56e18c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Heartbeat has been enabled which will allow us to check the uptime of
hosts and services.
Stack updates have been made to correct a couple templating issues and
reduce template sizes by using includes where we have common config.
Change-Id: I47e32ac4b4ce8ca3ea572d8384660011af7cde6a
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The elastic Stack has the ability to get application performance metrics
using the built in APM server. This change implements the APM server in
an existing ELK environment.
Change-Id: Ie6f533b81cfdb0c6a4ba2f33fd3b9f0a3e49a1fc
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
