The new option logstash_syslog_input_enabled has been added which will
allow users to enable a direct syslog input. When enabled, messages will
be processed via logstash and sent directly to elasticsearch.
Change-Id: Icb7712ecb8aae3d7f99df80ae1c5cd647a15ce83
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds Ubuntu 14.04 support to the project.
Change-Id: I20695e19409b63c6e1def4ccf8929c6d52be647e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds the scaffolding required to get multi-distro support
running in the roles. The change breaks up our playbooks converting all
of the tasks into various roles with internal dependencies. While this
will improve execution time, the change is being done to reduce boiler
plate and to allow us to build on the pattern used in OSA to provide
multi-distro capabilities.
A side effect of this change is a major improvement in idempotency. The
playbooks should now be 100% idempotent.
All of the templates have been left in the main playbook directory. This
was done to help ease the transition. In a future PR the template
structure will be moved into the roles where it needs to be.
The main variable files has been left intact. This file will be carved
up into role defaults in a future PR.
Change-Id: I938a10564128ce4078fa12edcf614dcdbd684b25
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
In order to reduce time to run and avoid complexities
from older environments that may have specific
requirements or devations from recommended practices,
disable the hardening play during the leap upgrades.
If security-hardening is desired, the recommendation for operators
is to run hardening after the upgrade has completed.
Change-Id: I85865a39e84e65ac211c1077f484bf008673c840
This was a provider specific command which can be
removed as it could remove unintended containers.
Change-Id: I179565f84fd8176cbcb79eacc8e63e0fef554223
The current variable, when there's only one host will result in 'false'
instead of 0, which is a jinja-ism however due to java-ism's "false" is
evaluated to 5 and that makes the index retention policy very wrong.
Change-Id: I2668e17c1cf15fe47842ff349ffa4f71c70257e5
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The fleet env.d config had a typo in it which would causing a failure
within OSA dynamic inventory. This corrects the issue.
Change-Id: I36016f4733a8800adbcce71abbb290ab4a24798b
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The logstash groks were running in line using the legacy method which uses
lexical sorting of all logstash filter files and loads them in order. While
this works it makes it so all data has to travel through all filters.
This change makes use of the logstash multi-pipeline capabilities
using a distributor and fork pattern. This allows data to flow through
logstash more quickly and not block whenever there's an issue with an
output plugin.
Finger-prints using SHA1 when there's a message and UUID when not. This
will ensure we're duplicating log entries which will help speed up
transations and further reduce the storage required.
Change-Id: I38268e33b370da0f1e186ecf65911d4a312c3e6a
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This allows alternate algorithms to be developed and enables the
same tasks to be included in a test to verify the planned retention
does not exceed the cluster storage capacity.
Change-Id: Ie3d80d6cfad16b946ccd790859bc7cd92b90fdef
* Implements G1 GC optionally. The variable `elastic_g1gc_enabled` has
been added with a default of false. If this option is set true and the
system has more than 4GiB of RAM G1GC will be enabled.
* Adds new thread options
* Better constraints coordination nodes
* Interface recover speed has been limited
* Buffer size is now set correctly
* Serialize elk deployment so that upgrades are non-impacting
Change-Id: I89224eeaf4ed29c3bb1d7f8010b69503dbc74e11
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Logstash is able to handle arcsight events, this PR enables that
capability.
Change-Id: Id220c671cc5d7cb7ee33fb53e2ae4185d579fc2a
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The beats were all enabled with the default settings to enable
dashboards and templating when the specific beat starts. In a large
scale environment this creates a DDOS against Kibana as all beats begin
uploading templates and dashboards clobbering one another. This change
moves the dashboard config into a common template and sets everything
using sane defaults so that we're not inadvertently killing our clusters
when rolling restarts happen, like in the event of an upgrade.
Change-Id: Ib48ea34a350335b72c3e3df941853c405072446a
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds the xpack monitoring capabilities to all of the core
beats we deploy.
Change-Id: Ib09388b83e18953cb180cdb93fec34e5917cc82c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds retention policy weighting based on experience with the
indexes in production in large scale clouds.
Change-Id: I0d09d4cfc68f70fe790170d5d54f1585616c5524
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
There was an error where the rention days was 2x greater than it was
supposed to be.
Change-Id: I9e9451e4381a32c0b2857eb75689561d3517b8d7
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
* Adds support for provisioning a Multi Node AIO using
CentOS 7.
* Cleans up older MNAIO/Compute/Infra image configs
* Increases LB/Logging/Swift VM ram to allow for CentOS rootfs
to load into RAM. (1GB to 2GB)
* Uses systemd-network networking for configuring network/bridges
* Adds keymap to kvm configuration to alleviate keyboard issues in
virt-manager
Change-Id: I54d903e7c1c70882e8b20a9cef4eafb42be46770
The variable elastic_thread_pool_size is expected to be an integer
however setting it as a fact and using it later results in it being a
string which creates unexpected results when the string has math applied
to it.
Change-Id: I339792c80378c86f785462a318a2565e946c7515
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
To ensure we're not creating too many logs files the logrotate config
has been added to truncate and compress log files every other day
with a max retention priod of 5 days.
Change-Id: I0dd85a334dba48e9fce3aad67ead867e2f46cb02
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds xpack monitoring by default for kibana, logstash, and
elasticsearch.
Change-Id: I03c8b8390d33bb996201671aae481415bfc1e691
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The ansible thread pool is supposed to be based on available processor
cores and not hyper-threads. This change updates our settings which will
allow the cluster to run generally better and give us more control over
JVM memory usage.
Change-Id: Ie19e31689c9b18ee2071b3e6fb8e725694fb7897
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Sharding should be reflective of the data-nodes within the environment.
Change-Id: I353e297ff2f952214fc8876c2330d690b6597ba8
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Allows for configuration changes to be redeployed on a
rebuild where previously it didn't attempt to update
the VMs configuration.
`
Change-Id: If14dbdfe7ba3e69a50127fa724ad3f2a8ed58c1a
