* modules/logstash/templates/kibana.vhost.erb: Need to allow POST to
_aliases and _search and OPTIONS to _search. Also fix the ProxySet
settings directive by placing it in a <Proxy> section.
Change-Id: I0bb11b72eec9e8753ed8cf1d5324f411bb574f49
* modules/logstash/templates/kibana.vhost.erb: Use mod_rewrite to proxy
elasticsearch so that we can safely proxy _mapping. To do this safely we
need to restrict requests to GETs. Also, add _nodes to the list of
proxies URIs.
Change-Id: Ibe86dc104c429c144b31fed547034ee14f7f2e1d
To allow for reuse of elasticsearch I'm splitting it away from
logstash. Also, I'm doing a hash check of the elasticsearch wget
for better security.
Change-Id: Iff42d538cd941abd50b000879ea4a237ea48d40e
* modules/logstash/templates/kibana.vhost.erb: Increase the connection
timeout to ElasticSearch when proxying to the limited ES API. This
prevents errors when ElasticSearch is slow due to cold caches.
Change-Id: I066a7c7822a0ae75bda52b9b37ad27bc07bd7e0a
Interesting read-only data on the elasticsearch cluster can be found at
_cluster/health, this opens that url in the apache reverse proxy so that
we can see the health of the cluster.
Change-Id: I8d7a2eeec1f02aaa78af5f348b1ae91af76f522d
* manifests/site.pp: Put elasticsearch nodes in a list variable that is
reused.
* modules/logstash/templates/elasticsearch.yml.erb: Update cluster
settings to be more appropriate for the larger cluster size, including
changing the minimum number of master eligible nodes that must be seen
before becoming operational in a cluster. Should help prevent split
brained clusters.
* modules/openstack_project/manifests/cacti.pp: Add new node to cacti
node list.
Change-Id: I9a18422aa56a56b9ad030892a312772d9b6b2a79
* modules/logstash/templates/elasticsearch.yml.erb:
Give 33% of memory to indexing instead of 40%.
* modules/logstash/files/es-logstash-template.json:
Flush the translog every 50k operations instead of the default 5k
(we can do 5k ops every second or two so the flushing was happening too
often). Limit the total number of shards per index per node to 3. We
have 5 nodes and 10 shards per index, 5*3 > 10 so we should be fine.
Set the field cache to soft type so that its entries can be garbage
collected in memory constricted situations.
* modules/logstash/manifests/elasticsearch.pp:
Apply es-logstash-template.json to the elasticsearch servers.
Change-Id: I2337fc41998fd00e090b0acfd29f007dfb6ec8df
* manifests/site.pp: List the two new elasticsearch nodes in the
appropriate lists.
* modules/logstash/manifests/elasticsearch.pp: Do not restart
elasticsearch when config files change. Service restarts are costly and
should be manually performed when necessary. Otherwise puppet should
simply update the config files.
* modules/logstash/templates/elasticsearch.yml.erb: Update elasticsearch
config with new cluster topology. Increase memory available for
indexing.
* modules/openstack_project/manifests/cacti.pp: Add new nodes to cacti
monitoring list.
Adding two more elasticsearch nodes to relieve memory pressure (more
nodes means fewer indexes per nodes which requires less memory to
manage). And two more nodes gives us more disk to retain older indexes
in. These new nodes should allow us to retain at least 3 weeks of
indexed logs.
Change-Id: I3a5a02311e939c8147e401110c7b96d085eb8274
Reviewed-on: https://review.openstack.org/36305
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
* modules/logstash/templates/kibana.vhost.erb: Set proxy timeouts and
allow /_status and /_search.
Change-Id: I67f6152bcda5999a27fef07c8428ac12d3a0201c
Reviewed-on: https://review.openstack.org/34252
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Use mutliple discover nodes to determine elasticsearch cluster
membership. Put a timeout on recovery starting instead of the default
to recovery immediately. Describe cluster topology in elasticsearch yaml
config so that it can make smarter decisions. Round robin kibana
requests across each discover node.
Change-Id: I08ef9dd158ddf6a6ce01dfb2050626f543d45b10
Reviewed-on: https://review.openstack.org/34106
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
We need to expand our elasticsearch install base. Update puppet to make
this possible.
Change-Id: Id0dae839b12ebf47715cf40a363832e0f661a94f
Reviewed-on: https://review.openstack.org/33910
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
It has been requested that we provide programmatic access to the
elasticsearch API in addition to the Kibana web UI. Do this by reverse
proxying http://logstash.openstack.org/elasticearch/$API_REQUEST to
http://elasticsearch.openstack.org:9200/$API_REQUEST.
The only values for $API_REQUEST that will be passed through are
_aliases, _status, and _search.
Change-Id: Ib41f6d91e2e59d493218074a67155af450ec8c93
Reviewed-on: https://review.openstack.org/33316
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Add a kibana module that installs kibana and flat configuration files,
and also disables the previous boring bland web frontend.
Change-Id: I772c66f07d2bc3a88128e101074be9d4162e3f8a
Reviewed-on: https://review.openstack.org/27089
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
This new logstash module adds classes to install logstash agents and
indexers as well as redis and elasticsearch. The configuration for each
of these services is rudimentary but it shouldn't be difficult to expand
the configs and make them useful.
Also, add a logstash.openstack.org node that will have an agent,
indexer, web frontend, redis, and elasticsearch installed on it.
Change-Id: I25b635f088f99d45cfaa70ed122c6433d3784937
Reviewed-on: https://review.openstack.org/19871
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
